Skip to content

data_import_log.md

Latest commit

 

History

History
67 lines (50 loc) · 5.52 KB

data_import_log.md

File metadata and controls

67 lines (50 loc) · 5.52 KB

Frappe Core Doctype: Data Import Log

The Data Import Log Doctype in Frappe is used to store detailed, record-by-record logs of data import operations. Each entry corresponds to a specific row (or a set of rows representing a single parent document with its children) processed during an import job initiated via the "Data Import" Doctype.

Core Purpose

  • To provide a granular audit trail for each data import attempt.
  • To record the success or failure status for each individual record processed.
  • To store error messages and tracebacks if a record fails to import.
  • To link back to the successfully created/updated document if the import for that record was successful.
  • To help users diagnose and troubleshoot issues with data imports by providing specific error details per row.

Key Files Defining Data Import Log

  • data_import_log.json: Defines the schema (fields, properties) of the Data Import Log Doctype.
  • data_import_log.py: Contains the basic server-side Python controller class.
  • data_import_log.js: The client-side JavaScript file for the Data Import Log form (which is minimal).
  • test_data_import_log.py: Contains unit tests.

data_import_log.json - Schema Definition

This JSON file outlines the structure for each Data Import Log entry.

  • Engine: "MyISAM": This is a notable choice, often selected for log tables due to potentially faster inserts when full transactional integrity for the log entries themselves is not the highest priority.
  • Key Fields:
    • data_import (Link to Data Import, In List View): A mandatory link to the parent "Data Import" document that this log entry belongs to.
    • success (Check, Default: 0, In List View): A boolean flag (0 or 1) indicating whether the processing of the corresponding record(s) was successful.
    • docname (Data): If the import for this record was successful, this field stores the name (ID) of the document that was created or updated.
    • row_indexes (Code, Options: JSON): Stores a JSON array of the original row numbers from the source import file that this log entry pertains to. This helps in tracing issues back to the specific rows in the uploaded spreadsheet.
    • messages (Code, Options: JSON): Stores a JSON array of informational messages, warnings, or user-friendly error messages generated during the processing of this record.
    • exception (Text): If a critical error (Python exception) occurred while processing this record, the full traceback is stored here for debugging.
    • log_index (Integer, In List View): A sequential index for the log entries within a particular "Data Import" job, useful for maintaining order.
  • in_create: 1: This property means that users cannot manually create "Data Import Log" records through the standard UI. These records are generated exclusively by the system during the data import process.
  • read_only: 1: Once created, Data Import Log entries are read-only, preserving the integrity of the log.
  • Permissions:
    • Only "System Manager" role has permissions to access (read, export, delete, etc.) Data Import Log records.
  • Module: "Core"

data_import_log.py - Server-Side Logic

The Python controller for the Data Import Log Doctype is very simple.

  • DataImportLog(Document) Class:
    • no_feed_on_delete = True: This class attribute ensures that deleting a Data Import Log entry does not itself generate a feed entry (e.g., an Activity Log), preventing excessive or recursive logging.
    • The pass statement indicates that there are no custom server-side methods (like validation hooks, on_update logic, etc.) specific to the Data Import Log Doctype. These logs are created and populated by the "Data Import" tool's Importer class (importer.py).

data_import_log.js - Client-Side Logic

The client-side JavaScript for the Data Import Log Doctype's own form view is minimal.

  • frappe.ui.form.on("Data Import Log", { // refresh: function(frm) { // } });
    • The file contains only a commented-out refresh function. This signifies that the direct form view of an individual Data Import Log document does not require any special client-side customizations.
    • These logs are primarily intended to be viewed in a tabular format within the parent "Data Import" document's UI (rendered by data_import.js) or exported for analysis, rather than being opened and inspected one by one as forms.

How It's Used

  1. A user initiates a data import via the "Data Import" Doctype.
  2. As the Importer class processes each record (or a small group of records like a parent document and its children) from the uploaded file:
    • A new "Data Import Log" document is created programmatically.
    • It's linked to the parent "Data Import" job via the data_import field.
    • The success field is set based on the outcome of processing that record.
    • If successful, docname is populated.
    • Relevant row_indexes, messages (e.g., validation warnings), and exception (if an error occurred) are recorded.
  3. These log entries can then be viewed by the user within the "Data Import" document's interface (in the "Import Log Preview" section) to understand the status of each part of their import and to diagnose any failures.
  4. The "Data Import" tool also provides an option to "Export Import Log," which would compile these individual log entries into a downloadable file.

The Data Import Log Doctype is essential for providing transparency and traceability for bulk data operations in Frappe.