Import Data

Overview

Import Data (Import Mapping internally) is the menu for importing a CSV file to update the profile of friends who already exist. It is update-only — it never creates a new user from the file, and never removes anyone.

The heart of this menu is the match key — you have to tell the system which field on the friend profile (LINE UID / mobile number / email / custom attribute) matches which column in the CSV, so each row can be tied to a specific friend. Rows that can't be matched are skipped, not turned into new friends.

The import doesn't run inside the request — cms-api only records the job as pending and publishes it onto RabbitMQ, and line-management-worker-go is what actually updates the database.

Who uses it — CMS admins holding customer data from an external system who need to sync those values down onto many friend profiles or custom attributes at once.

User journey

1. Review the import jobs

Open the Import Data menu. The table shows File, Status, Records, Created, Import log, newest first.

• There are 5 statuses — Pending / Processing / Completed / Completed with errors / Failed.
• The Records column changes meaning with the status: while pending/processing it shows {n} rows taken from the validation snapshot saved when the job was submitted, but once finished it switches to Updated {x} · Skipped {y} from the real result the worker wrote back.
• While any job is still pending or processing, the page refreshes the list itself every 10 seconds, and stops once everything has finished.
• The Import log column only carries a download link for jobs that actually skipped rows.

2. Walk the 4-step import wizard

Press Import CSV.

Step 1 — Upload the file

Drag in a CSV (max 20 MB). The frontend checks the size before sending, then uploads to storage; the system reads back the columns, the first 10 sample rows, and the total row count.

Step 2 — Preview the data

A table shows the first 10 rows with a Showing the first {n} of {m} rows line, so you can check the file was read as expected (especially columns that may have shifted because of quotes or commas inside the data) before configuring the mapping.

Step 3 — Map the fields

1. Pick the match key — on the left, a field on the friend profile (LINE User ID / mobile number / email / any custom attribute declared in Attribute Setup); on the right, a column in the CSV.
2. Map the remaining columns to target fields — the options are split into two groups (Profile fields / Custom attributes), and any column can be set to "Don't import".
3. The column chosen as the match key is automatically removed from the mapping list and shows a "Match key — used to find the friend" tag instead of a select.
4. Each target field can only be used once — once a column is mapped to a field, that field is disabled in every other column's list.
5. Next becomes available once all three are true: a match key field is chosen, a match key column is chosen, and at least one other column is mapped.

Step 4 — Review and start the import

1. Press Next and the system runs a dry-run validation against the real file, then shows four summary cards — Total rows, Will update, No match — skipped, Invalid — skipped.
2. If any rows will be skipped, a warning box states the count and a Skip reasons table lists the row count per reason.
3. Press Start import. The system creates a pending job, queues it, and returns to the list so the status can be tracked.

3. Check the result and fix skipped rows

1. Wait for the status to become Completed or Completed with errors (the page refreshes itself every 10 seconds).
2. If rows were skipped, press Download log to get a CSV containing the entire original header plus an appended Error Reason column, holding only the skipped rows with every reason that applied to each.
3. Fix the file according to those reasons, then import only the corrected file — re-importing the whole original file just writes the same values again; nothing checks whether a file has already been imported.

Key details verified against the source

Confirmed from internal/modules/importmapping/{controller,service,dto}.go in line-management-cms-api-go and internal/importmapping/{service,types}.go in line-management-worker-go:

• LINE User ID can be a match key but can never be written to. The worker keeps two separate column lists: the set allowed as a key (which includes user_id) and the set allowed as an update target (which deliberately excludes user_id, with a comment stating that the UID is the LINE identity link and may only serve as a read-only match key). In the other direction, the display name is a valid target but not a valid match key — the two option lists are genuinely different.
• The step-4 numbers and the real result can disagree, because "validation" and "execution" use different rules in two places. This is the detail that affects users most in this menu:
  • Mobile-number matching — the worker strips spaces and dashes from both the CSV value and the database value before comparing (so "081-234-5678" does match "0812345678"), while cms-api's validation compares the strings directly after only trimming. A row using a phone number as its match key can therefore be reported as "No match" in step 4 and still be updated successfully when the worker runs.
  • Email format checking — cms-api uses a stricter regex; the worker uses a looser rule (just an @ and a dot in the domain). Some rows are counted as "Invalid" during validation yet pass and get updated for real.
  • The validation result should therefore be read as a conservative estimate, not an exact forecast.
• The numbers in the "Skip reasons" table can add up to more than the number of skipped rows — by design. A row with both a malformed email and no matching friend is counted under both reasons, while the "No match" and "Invalid" cards are kept mutually exclusive so the four cards still sum exactly to the total row count.
• The bucketing priority is explicit — a row with no matching friend always counts as "No match" even when its data is also malformed (no amount of data fixing makes it importable if that friend doesn't exist). "Invalid" is reserved for rows that did match (or can't be matched at all, because the key is empty) but carry bad data.
• Validation uses a single database read for the whole file — cms-api pulls every non-empty key value for the OA's friends into an in-memory lookup set first, then walks the rows, with no per-row queries. A ten-thousand-row file validates quickly, but the result is a snapshot of the friend base at that instant.
• Empty cells never blank existing data — the worker skips cells that are empty after trimming and never writes them, so importing a file with gaps can never wipe a friend's existing data. If you actually need to clear a value, edit it per person on the All friends page.
• Custom attributes are merged, not replaced — the worker writes with the jsonb merge operator (custom_attribute = COALESCE(custom_attribute,'{}') || {new values}), so keys not present in the file are left intact.
• Duplicate keys in the database mean "first one wins" — with no defined winner. When building the lookup, the worker keeps only the first row seen per key value, and the query has no ORDER BY. If two friends share an email address, there is no guarantee which of them gets updated. Choose a unique match key (LINE UID is the safest).
• Duplicate keys in the CSV mean "last one wins" — the worker processes rows in file order, and every row matching the same friend overwrites the previous one, so the surviving value comes from the bottom-most row.
• The "Failed" status does not always mean the file was broken. The worker derives the status from the real outcome: updated > 0 with no skipped rows → success; updated > 0 with skipped rows → partial; everything else (including a perfectly readable file where not one row matched) → failed. Open the log file before concluding something is broken.
• There is a duplicate-work guard at the database level — the worker "claims" the job with a conditional UPDATE ... WHERE id = ? AND status = 'pending'. If no row is affected (job missing, already claimed, or finished) it acks and drops the message, so a duplicate queue delivery never applies the data twice.
• Every failure after a successful claim finalizes the job as failed and acks — it never leaves a row stuck at processing, and never requeue-loops.
• If a mapped column has disappeared from the file's header, that mapping is silently skipped — the worker logs a server-side warning and carries on with the remaining mappings, without failing the job (unlike a missing match-key column, which fails the whole job).
• The source file is read three times from three different places — at upload (cms-api, to build the preview), during step-4 validation (cms-api, re-reading from storage), and when the worker runs (from storage). All three strip a BOM before parsing (the worker's comment notes that our own CSV exports carry one for Thai Excel, and without stripping it the first header cell would never match a configured column name).
• File ownership is enforced by inspecting the path prefix — both the validate and the create-job paths require filePath to start with private/<lineOaHash>/imports/mapping/, or they answer 400. Even a guessed filename from another OA can't be submitted for processing.
• The stored filename is sanitized and time-prefixed — anything that isn't a letter, digit, ., - or _ is replaced with _, and an epoch-millisecond prefix is prepended. A Thai filename therefore becomes a run of underscores in storage (the original is kept in the file_name field, which is what the table displays).
• The 20 MB ceiling is checked three times — in the browser before sending, in the controller from the size declared in the multipart part, and again from the number of bytes actually read (guarding against a declared size that doesn't match reality).
• The accepted file types are broader than what the UI offers — the controller accepts a .csv extension or a content type containing csv or text/plain, so an extension-less text file still gets through when calling the API directly.
• The validation snapshot stored on the job is "what the user saw and confirmed", not a recomputed value — the frontend attaches the step-4 result to the create-job request and the backend stores it into the validation column as sent (only when it isn't null). The row-count figure shown in the table while a job is still running comes from this value.
• If queueing fails, the job row has already been created. The backend inserts the import_mapping_job row first, then publishes to the queue; a publish failure answers 500 but does not roll back the insert. The result is a job stuck at pending forever with nobody to pick it up (and there is no delete button in the UI). A job sitting at pending unusually long while the worker is healthy is the signature of this case.
• The reason ordering in the real log differs from the validation screen — cms-api's validation emits reasons in a fixed order (empty key → bad email → bad phone → no match), while the stats the worker writes back are sorted by count descending then alphabetically. Don't compare the two lists line by line.
• The log download link is a presigned URL regenerated on every list load, and the column is empty (—) when no row was skipped.
• A failed log upload does not fail the job — the data updates already happened, so the worker records the error server-side and finalizes the job without a downloadable file.

Related menus in this journey

• Member Database — a completely different mechanism, not just a different menu. See the comparison at the end of this section.
• Settings > User Attribute Setup — the attributes declared there are the source of the custom.<key> options in both the match key and the target lists (walking child attributes, and only for masters with status active). With no attributes declared, the lists offer only the standard profile fields.
• All friends — where you verify the per-person outcome after an import finishes, and the only place a field can be cleared for an individual (an empty cell in an import never clears anything). Note too that the CSV import in that menu's "Import history" tab is a different mechanism from this one — it accepts only a Line UserID column to create/reference records, with no field mapping.
• Segment audience — values brought in by this menu are immediately usable as target-group conditions once the job finishes.
• Rich message / Campaign — the fields imported here are exactly what merge tags in message content read from, so importing them completely before sending reduces the number of recipients with incomplete merge-tag data.

How Import Data differs from Member Database

Every difference below is verifiable in the source of both modules:

	Import Data (this menu)	Member Database
Destination tables	line_user — profile fields and custom_attribute of existing friends	customer_database / customer_database_row, its own tables
How identity works	A match key must be chosen and matched to a CSV column	None — every row is stored as it appears in the file
Effect on friend profiles	Overwrites existing values on matched friends	Never touches a friend profile
Field mapping	Every column is mapped to a target by hand	None — column names are field names verbatim
Who consumes the data	Every menu that reads friend profiles (All friends, Segment audience, campaign merge tags)	Form builder, to check whether a respondent is a real member
Processing	Queued to RabbitMQ for a worker to run in the background	Immediate, in a single request (transactional)
Unusable rows	Skipped individually and summarized in a log file	Cause the whole file to be rejected
History / deletion	Kept as import-job history; no deletion	Deletable (soft delete), unless a form is bound to it

Put as briefly as possible: this menu changes the friend data itself, while Member Database adds a reference table alongside it without touching friend data.

Things to watch out for

• Don't read the step-4 numbers as the final answer — validation and execution use different rules for phone matching and email format checking (see the details above). The "Will update" figure is usually lower than reality when a phone number is the match key.
• The step-4 result is a snapshot of that moment, not a data lock — if friends are added or removed while the job sits in the queue, the real numbers will differ.
• There is no cancel after pressing Start import, no way to delete a job, and no per-job detail page. Everything visible is the history table and the log file.
• Overwritten values can't be recovered — the system stores the previous value nowhere at all. If a field is mapped wrongly and imported, the only remedy is to import a corrected file over the top.
• Always choose a unique match key — if several friends share a phone number or an email address, there's no guarantee which one gets updated.
• If the CSV has several rows pointing at the same friend, the surviving value comes from the bottom-most row — de-duplicate the file first.
• A Failed status may just mean "not one row matched", which usually signals a wrong match key or column — open the log file first.
• A job stuck at Pending unusually long while the worker is healthy may be one whose row was created but whose queue publish failed — that needs a system-side fix; nothing in the UI can resolve it.
• Empty cells don't clear existing values, so this menu can't be used to wipe data.
• The log download link expires (presigned URL) — click it from a freshly-loaded list.

Access control (verified against the source)

• Frontend — wrapped in a PermissionGuard checking the VIEW action on the import-mapping subject.
• Backend — all 5 routes are wrapped in a real, enforced ModuleGate(d, "import-mapping"), with the same middleware instance attached to each route individually (POST /import-mapping/upload, GET /import-mapping/fields, POST /import-mapping/validate, POST /import-mapping, GET /import-mapping). If a platform admin closes this module for an organization, customer-type users in that org get a 403 immediately even calling the API directly (onemoby-type callers always bypass the gate). The module name is the same string on both the frontend and the backend here.
• Every route requires all three pieces of OA context — lineOaId, organizationId and lineOaHash (the value obtained when the LINE OA is selected after login). Missing any one of them returns 400 "Line OA context is required" before the database or storage is touched.
• There is a second cross-tenant guard at the file-path level — a submitted filePath must start with the caller's own OA prefix (private/<lineOaHash>/imports/mapping/) or it answers 400. Module-level permission alone is therefore not enough to have another OA's file processed.
• Job listing and every key lookup are always scoped by the line_oa_id + organization_id from context, and rows carrying a deleted_date are excluded.
• ModuleGate is designed to fail open — if the caller can't be identified, no organization is in scope, or the config lookup fails, the request is allowed through (and logged).
• Action-level policy annotations (create / read / readAll) are declared on every route but run through the same no-op middleware as other modules — the real enforcement is the module-level ModuleGate.
• There is no route at all for deleting or editing an import job (only upload / validate / create / read) — the import history is append-only.

Source file reference

Frontend (cms-web-2026)

• src/app/import-mapping/page.tsx — entry point and PermissionGuard
• src/components/import-mapping/import-mapping-list.container.tsx — list fetching and the 10-second auto-refresh while jobs are active
• src/components/import-mapping/import-mapping-table.tsx — all 5 status tags, the Records column that changes meaning with status, the log download link
• src/components/import-mapping/import-wizard.container.tsx — the 4 steps, the Next-button conditions, the invalidation of later steps when the file changes
• src/components/import-mapping/steps/upload-step.tsx — the in-browser 20 MB check
• src/components/import-mapping/steps/mapping-step.tsx — match key selection, option grouping, the "each target once" rule
• src/components/import-mapping/steps/review-step.tsx — the four summary cards and the skip-reasons table

Backend (line-management-cms-api-go)

• internal/modules/importmapping/controller.go — 5 routes, ModuleGate("import-mapping"), file size/type checking
• internal/modules/importmapping/service.go — CSV parsing, storage writes, the dry-run validation logic and reason bucketing, job creation and queue publishing
• internal/modules/importmapping/dto.go — the lists of usable key/target fields and cms-api's email/phone regexes

Worker (line-management-worker-go)

• internal/importmapping/service.go — the atomic claim, the key lookup build, per-row updates, the jsonb merge, log file creation and status derivation
• internal/importmapping/types.go — the key-usable vs. writable column lists (where user_id is excluded), the reason list, the stats structure
• internal/importmapping/consumer.go — binding the handler to the queue

See the documentation team's earlier technical write-up at Import Mapping