Direct Data Export v2 is an advanced feature that provides automated, secure, and highly granular access to your employee engagement data. It creates a seamless, server-to-server transfer of raw data directly to your chosen file storage location.
This solution is ideal if you need to integrate raw Firstup data into your own data warehouse, enabling advanced reporting, detailed ROI analysis, and event-triggered engagement strategies at scale.
There are two ways to gain access to this raw data:
- Via SFTP directly to your organization's Secure File Transfer Protocol (SFTP) location
- Via API directly to your chosen file storage location
Note: If data anonymization is enabled for your community, direct data exports will not include raw Personally Identifiable Information (PII). If a full export of existing data is needed, please contact your Firstup representative.
Before You Start
- This article assumes you are signed into Creator Studio, have a role with permissions to access this feature, and have an understanding of Firstup insights.
- You have an understanding of SFTP (Secure File Transfer Protocol) or API connections.
- For an SFTP connection, you have a nominated contact in your organization, contact details, username, password or PEM key, server name, port and folder location for a continuous automated data export. You are responsible for providing correct and secure credentials.
- For an API connection, you have gained an authorization token to access the API curl command, with the data_exports.read scope applied. This token can be either oauth as a user or oauth as a server.
- Refer to Files for a high-level summary of the file types available.
- Refer to the Files Metadata for a full breakdown of the metadata available.
- Exported file types may include CSV, XLSX, or JSON.
Access Automated Direct Data Export via SFTP
Your automation requires an initial setup. To do this:
- Navigate to Configure > Insights > Recurring Data Connections.
- Select Set Up SFTP.
- Set your Start Date, End Date (optional), and the Frequency of your automated data export.
- Enter Your Server link and Port details. For example, https://demo.com is your server link, and your port number is 22.
- Enter the Destination Folder and the Username of the location receiving the SFTP file.
- Enter your password or PEM key.
- Enter the Contact Email. This registered email address receives all failed or successful delivery reports.
- Select Save.
If your setup is correct, an SFTP Connection Active notification appears. If our SFTP check identifies connection issues, you receive a notification.
Note: The progress of automated data exports is not shown until complete. You can find completed export jobs (successful or unsuccessful) from the Event History tab.
Pause SFTP
You can pause your automated SFTP connection anytime. Pause your connection to, for example, edit the frequency of your export.
There are two options:
- Pause After Transfer - This option lets the current transfer job be completed before pausing to the next scheduled job.
- Pause Immediately - This option pauses any job running in the background.
Note: Once you have set them up, the server link, port, and destination folder are immutable. If you need to change any of these fields, delete and restart the entire SFTP connection.
Access Automated Direct Data Export via API
Alternatively, you can create a recurring data connection via API. This requires an initial setup in Creator Studio before you can access the data by API request.
- In Creator Studio, navigate to Configure > Insights > Recurring Data Connections.
- Select Set Up API.
- Select the Start Date, optional End Date, Frequency and Contact Emails.
- Select Save.
Once you connection is setup, you can perform API requests to access data.
Note:
- The connection looks for data at your selected frequency. For example, if you select every 4 hours, there will only be data to access during this time period.
- If there is no activity to capture within the time period, your API request will return an empty payload.
- The API request accesses your data, it does not produce the data file. The generated url provides you with a link to a zip file.
- There may be an initial lag for data request after your initial connection setup.
- You need to request an API token from your Firstup representative.
For guidance on how to make an API request, please refer to our Developer API Documentation Portal, for API Reference and Documentation on the three available API requests:
- Get the latest export file for a given type (a given type is one of users, activity or campaigns)
- List all available export files for a given type
- Get metadata and download URL for a specific export file
For further guidance, refer to our Supporting API User Documentation on these endpoints.
Historical Data Request
To catch any data that exists before your automated data export start date, you can request this outside of the automation. You can select up to 24 months worth of data from the date you select Submit.
This data export can take some time. Our SLA for a successful export is 7 days. You can view the export's progress from the Event History tab.
Event History Tab
Here you find an audit log of all export files created and transferred to your SFTP or API destination, including automated and historical requests.
The export files shown in the log are those generated, delivered, and failed delivery attempts.
The log includes the file name, start time of the job, file size, status (successful or unsuccessful) and the SFTP destination.
Failed delivery attempts can be retried from this section by selecting the button in the status column.
Note: To diagnose the reason for any failed deliveries, access the Activity Log from the Configure screen.
Files
The types of data that are included in the files are:
- User-level data - User profile details, attributes, and account status.
- Campaign-level data - Campaign metadata, delivery channels, author information, targeting, and more.
- Activity-level data - Raw event logs including clicks, opens, likes, video plays, acknowledgements, etc. These are indexed by event timestamp.
All files are delivered as zipped packages to your SFTP location. The file naming format is:
<programID>-<mm-dd-yyyy-hh>-<file type>-<export type>.gz
Note:
- <file type> is replaced with the type of data in the file: <campaign>, <user>, <activity>
- <export type> is replaced with the type of export: <historical>, <incremental>
Files are compressed using .gz format. Once the file is uncompressed, they are in .csv format.
File Metadata
The folder can contain up to three files consisting of updated records of users, campaigns, and user activities, respectively. The presence of each file is dependent upon the updates for that day. For example, if no campaigns were created or updated that day or if there wasn't any user activity, those files would be absent from that day's folder.
Clicking on a folder in the folder list will open its contents, where all files can be downloaded in bulk or individually. Files are available for download for seven days before they expire from the system.
Note: Please refer to the Glossary of Terms for more details on the data columns listed below.
User-level
Includes users whose demographic data has been updated within the set time frame (the frequency of your data export).Data columns included:
- User ID
- First Name
- Last Name
- Universal Identifier
- Experience Status
- Blocked At
- Studio Role
- Last Visited Date
- Account Created Date
- Registered Date
- Preferred Name
-
Standard Attributes
- profile_zip
- profile_job_title
- profile_birthdate
- attribute_company
- attribute_department
- attribute_job_description
- attribute_gender
- attribute_city
- attribute_state
- attribute_country
- attribute_region
- attribute_hire_date
- attribute_start_date
- attribute_language
- attribute_work_location
- attribute_timezone
- attribute_locale
- attribute_cost_center
- attribute_division
- attribute_manager
- attribute_employee_type
- attribute_business_unit
Campaign-level
Includes all campaigns that have been newly created or updated within the set time frame (the frequency of your data export).Data columns included:
- Campaign Name
- Campaign ID
- Publish Date
- Archive Date
- Publisher User ID
- Publisher Name
- Author User ID
- Author Name
- Content Version
- Studio URL
-
Topics
- Topic ID
- Topic Name
- Topic Subscriber Count
- Initiatives
- Is Featured
- Is Shareable
- Is Commentable
- Is Translatable
- Duration
- Priority
- Acknowledgeable
- Author Alias ID
- Author Alias Name
- Publication State
- Source Type
- Audiences
- Content Type
- Channels (Delivery Channels)
- Acknowledge Message
- Resources
- For You
- Optimized Delivery
- Automatic Retargeting
- Subject Line
- Preview
- Generated At
Activity-level
Activity data includes raw events, rather than hourly (UTC) aggregated engagement data that is available in Direct Data Export 1.Data Columns included:
- Content ID
- Content Name
- Content Published At
- User ID (user ID of the user who engaged with the content)
- User Name (name of the user who engaged with the content)
- Likes
- Shares
- Bookmarks
- Email Opens
- Feed Clicks
- Emails Delivered
- Emails Clicks
- Push Clicks
- Impressions
- Downloads
- Poll Responses
- Acknowledge Counts
- Platform
- Date (date of engagement)
- Assistant Delivered Count (Now refers to Notification Center)
- Assistant Opens (Now refers to Notification Center)
- Video Plays
- Push Delivered
- Link Clicks
- Link Clicked
- Opens
- Clicks
- Subject Line
- Preview
- Generated At
FAQ
How is DDE2 different from DDE1?
- Automated SFTP data transfer has a one-time setup, allowing a forever data integration.
- Faster data delivery cadence, avoiding the daily increments.
- True, raw event data that includes every single detailed user activity.
Does DDE2 provide real-time data?
No. DDE2 is near-real-time (as frequent as every four hours) but not a streaming or API-based real-time service.
What happens to DDE1 once DDE2 is out?
Both products will exist for the foreseeable future. DDE1 will remain inside Insights under the Download section, whereas DDE2 is accessed through Configure > SFTP Configure.
How soon will the files be delivered to client folders?
Regular recurring incremental files will be delivered to the target SFTP folder within up to 1 hour after the accumulation period (i.e., every 4 hours/daily/weekly/monthly). Fulfillment of one-time Historical data export can take up to 7 days.
What if I need the Firstup IP?
We will provide our IP address for corporate IP whitelisting.
Can the configuration be updated?
SFTP config fields are immutable once set up. Only the frequency, start date, and contact email can be updated if the integration is in a stopped state. For other changes, the setup must be deleted and recreated.
What if a file transfer fails?
Failures are logged in the DDE2 audit table with status, time, file size, and SFTP info. Firstup Super Admins can retry failed deliveries within seven days. A checksum file is generated for each export to validate integrity.
What can Firstup Super Admins manage for DDE2?
- Write access to all pages in DDE 2
- Can create, pause, and stop SFTP connections
- Monitor file transfer statuses
- Retry failed file deliveries
What can Brand Super Admins manage for DDE2?
- Read access to all pages in DDE2
- Monitor file transfer statuses
What about security?
Only secure SFTP transfers are supported. Clients are responsible for providing correct and secure credentials. No passcodes are required for file extraction once downloaded.
Comments
0 comments
Article is closed for comments.