Release Notes

FactoryTX-core Release Notes

v3.9.0 (2024-07-25)

Highlights

  • Infor ION API Receiver - Added new receiver to support exporting data tables from an Infor ION system

Bug Fixes and Optimizations

  • Updated pymssql version to fix SQLServer query errors

v3.8.2 (2024-06-27)

Bug Fixes and Optimizations

  • All Non-Historial Data Receivers - Save DataFrame on Transform/Transmit failures

  • GE Historian Receiver - Fix issue with restream not resetting state

  • OPCUA Receiver - Fix issue where a transform error would stop dataflow

  • File Receivers (all) - Fix Excel Parser having invalid state when using deduplicate_rows option

v3.8.1 (2024-06-06)

Bug Fixes and Optimizations

  • GE Receiver - Handle invalid polling/data-sample interval configurations

  • OPCUA connection issues - Client session/connection stability

  • Optimization for config validation and streams UI (Performance gains up to ~40x faster validation and streams viewing)

v3.8.0 (2024-05-01)

Highlights

  • SQLReceiver - Add universal query timeout

  • Azure KeyVault - Refresh vault secrets on config validation

  • FactoryCONNECT - Workflow UI - Add “FactoryTX Demo” feature toggle to apply for “test_all” function

  • FactoryCONNECT - Workflow UI - Optimize “test_all” call

Bug Fixes

  • IoTHub Transmit - Fix payload size not being calculated correctly

  • SQL Receiver - Recycle DB connection pool on query timeout or database connection error/restart

  • GE Historian - Fix test_connection call for showing in FactoryCONNECT

  • Azure KeyVault - Fix paging for proxy addresses

v3.7.0 (2024-04-03)

Highlights

  • Add generic Normalize Transform

  • Add Snowflake Receiver

Bug Fixes

  • GE Receiver - handle unexpected bool values

  • GE Receiver - handle payload data level errors

v3.6.1 (2024-03-11)

NOTICE

Highlights

  • Added support for Nested Azure IoTEdge Devices In-Platform

  • Added support for Azure EventHub Transmit

  • Added support for demo test connections in workflow UI

  • Added marker errors for FTX device high memory usage and low disk space

  • Enhanced SQL Connection Workflow UI - made Database URL and Individual Fields switchable and set default port based on database type

  • Added query times to log messages for SQL Receiver

  • Added topic field missing warning to log messages for transmits

Bug Fixes

  • Fixed IoTHub Receiver workflow UI

  • Fixed PiWeb connection workflow UI

  • Fixed ChecksumRecord Transform to handle data with iterables (e.g. list, set)

v3.5.0 (2024-02-08)

Highlights

  • Added IoT Hub receiver to connection workflow

  • Added Enact receiver to connection workflow

  • Added GE receiver to connection workflow

Bug Fixes

  • Fix bug that caused file receivers to create massive amount of empty directories

v3.4.0 (2024-02-01)

Highlights

  • Added open weather receiver to connection workflow

Bug Fixes

  • Fixed connection test issues in connection workflow

v3.3.0 (2024-01-29)

Highlights

  • Added remaining SQL database type to connection workflow

  • Added mqtt receiver to connection workflow

Bug Fixes

  • Fixed markers.db file missing preventing factorytx_web from starting

  • Fixed bugs in test_all API endpoint

  • Fixed test_connection method for oracle

  • Fixed factorytx_web crashing/hanging occasionally when submitting config

Other

  • Documented adding env variables to factorytx-cloud

v3.2.0 (2024-01-25)

Bug Fixes

  • Fixes for test connection API

v3.1.3 (2023-12-15)

Bug Fixes

  • Fixed issue with race condition on new instances where FTX would not start correctly

v3.1.2 (2023-12-15)

Bug Fixes

  • Fixed issue with MQTT receiver where it would not re-subscribe to a topic after a connection error

  • Fixed issue with FileReciever deduplicate_rows option creating the row hash where it was not needed

v3.1.0 (2023-12-14)

Highlights

  • Added connection workflow in MA for FileReceiver, OPCUAReceiver, SQLReceiver, and PiWebReceiver

  • FTX devices that are linked to a MASE are now viewable in Grafana dashboard (https://grafana.aks-multi-tenant-prod-westus2-kube.sightmachine.com/d/JUdMZOnSz/factorytx-instance-overview)

  • Added support for using secrets from Azure KeyVault in configurations

  • FactoryTX processes are now run inside of a python virtualenv (users are able to install custom python packages in Notebooks without interfering with FTX processes)

  • Updated JupyterLab to v3.6.6 (able to upload larger files, among other fixes)

  • Fixed container security issues (CVEs flagged by GCR) by updating to Debian Bookworm (v12)

  • Added file receiver row-hash de-duplicating option for parsing files that change out of row-order

Big Fixes

  • Fix SQL receiver validation not catching invalid database_url format errors

  • Optimizations around having invalid FTX configurations or websocket tunnel connections

Other

  • Added FTX-Terminal control commands to docs

  • Added JupyterLab bash audit log

  • FTX is now paused for new instances by default

  • Updated websocket tunnel to v1.1

v3.0.0 (2023-09-07)

NOTICE

  • As of v3.0.0+ FTX is not able to be accessed from the host machine, this is to facilitate the new in-platform functionality via the websocket tunnel connection to the Sight Machine MA platform

  • The configuration editor has also set to read-only mode on the FTX frontend, the only way to edit the configuration now is through the new MA draft configuration editor

  • Additionally, ‘sight_machine_cloud’ has been removed from default config, this must be configured via environment variables

Highlights

  • Added ability to add extra hosts entries to /etc/hosts via the configuration file

  • Added Prometheus metrics for various receivers to gauge health/performance

  • Report DeviceEnvironment in status reporting endpoint (for MASE FTX Devices Table)

  • Updated Balena deployment script to use a GCR image directly (deployment time down from ~20minutes to ~1 minute)

Bug Fixes

  • Fixed issue with GE Historian markers for ConnectionError not getting cleared correctly

  • Various documentations updates and fixes

v2.30.0 (2023-07-10)

Highlights

  • Prometheus support for the following: IoTHubTransmit, RDPV3Transmit, SMB, S3, MQTT, FileReceiver, EnactReceiver

  • Provide DeviceEnvironment in status reporting endpoint

v2.29.0 (2023-06-20)

Highlights

  • Update Documentation for API Keys, receivers, RDPv3 Transmit examples

  • Azure IoTHub Transmit - Use Edge Environment Authentication

  • Azure IoTHub Transmit - OPC Publisher Format

Bug Fixes

  • Fixed bug with IoTHub Receiver where ‘start_datetime’ was required to work

  • Fixed bug where FTX cloud settings were not loadable without an FTX configuration

  • Fixed CircleCI UnitTest threading issues causing false failures

v2.28.1 (2023-06-02)

Bug Fixes

  • Make RDPv3 re-check DNS after an error occurs

v2.28.0 (2023-05-30)

Highlights

  • Added SCE GreenButton XML Utility Data Parser

  • Added BigQuery support for SQL Receiver

  • Added a push job for the MA platform to receive FTX device status updates.

  • Updated balena script to support deploying multiple customer modules to one device

Bug Fixes

  • Fixed bug where dynamic filters created by FTX_CREDS would build up and not get cleaned.

  • Fixed bug where timestamp would not advance if GE Historian Receiver received only bad quality data

  • Fixed bug where state was not being saved when starting/stopping FTX

  • Fix CircleCI build-related issues (web pack, ADL creds)

v2.27.0 (2023-04-25)

Highlights

  • Added GE Proficy Historian API Receiver with support for both Interpolated and Raw data sources.

  • Improve documentation for configuring environment variables.

Bug Fixes

  • Fixed issue where Enact receiver’s configuration refresh_token was overwriting state even if it wasn’t changed.

  • Fixed SMB receiver not using the ‘port’ configuration option.

v2.26.1 (2023-04-14)

Bug Fixes

  • Fixed an issue where CircleCI builds were breaking after test servers were added for unit tests.

Upgrade Notes

  • Existing customer repositories will need to run the setup_circleci_config.sh script in order for CI builds to work.

v2.26.0 (2023-04-11)

Highlights

  • Added support for config variables when configuring environment variables (See “Configuration Reference” in docs for more details)

  • Updated samba from 4.17.6 -> 4.18.1

Bug Fixes

  • Fixed issue where azure iot and mqtt transmits were pausing after each run.

v2.25.0 (2023-03-31)

Highlights

  • Added MQTT receiver w/ support for sparkplug protocol.

Bug Fixes

  • Remove misleading error message on smb receiver saying a config was missing.

v2.24.2 (2023-03-10)

Bug Fixes

  • Fix a regression bug that would cause balena deploy.sh to fail.

v2.24.1 (2023-03-10)

Bug Fixes

  • Use more stable samba url link. (CircleCI builds with the previous one were failing since it no longer existed)

  • Fixed issue where nginx and azure cert handler were failing to run

  • Update iothub, mqtt, and edgehub transmit docs.

v2.24.0 (2023-03-01)

Highlights

  • Added websocket tunnel support for azure iot edge (including nested edge)

  • Added parquet parser to file-related receivers (localfile, s3, etc.)

  • Added Azure Data Lake Gen2 receiver

Bug Fixes

  • Fix issue where opcua nodes with delimiter characters were throwing errors

  • Fix cleanup task for file-related receivers where completed files were only being removed at the top level.

v2.23.0 (2023-02-21)

Features

  • Added support for smb v3

  • ‘skip_parser_errors’ option added to file receivers (Only works for csv files)

Bug Fixes

  • Fixed issue where azure deployments with the same branch name would not take effect.

Upgrade Notes

  • If migrating smb from v2 to v3, “netbios_domain” needs to be configured. If any issues arise, please post in the #factorytx channel.

  • Voltron-related: Routing changes in the application have been made. In order for the FTX web and jupyter notebook to function correctly, FTXSE needs to be updated to 2.1.4

v2.22.0 (2022-12-02)

Features

  • Added ‘iothub’ receiver for azure iot edge

  • Added ‘iothub’ transmit for azure iot edge

  • Added deployment script for azure iot edge (See scripts/az_iot/README.md for details)

  • Added public documentation page: https://factorytx-docs.sightmachine.com/. The docs for each release will be included in the link.

Bug Fixes

  • Fix WSGI-related bug where prometheus endpoint was not working, causing alerts not to show up.

  • Updates to library and package, mainly to resolve failing CI tests

v2.21.0 (2022-12-02)

Highlights

  • Move nginx to factorytx-core container (For azure iot edge)

  • Some python library upgrades (flask, requests, pytest, and more)

Features

  • Added support for historian in OPCUA receiver

Bug Fixes

  • Fixed identifier type issues with pubsub where only String would work

Upgrade Notes

  • If you are running into pylint issues in CircleCI, update the current CircleCI config files by checking out this release of factorytx-core(or newer), deleting the existing config files, and running the setup_circleci_config.sh script to recreate it.

  • If migrating custom OPCUA historian receivers with opaque: True, you will have to create a new receiver and set “default_identifier_type”: “ByteString” Additionally, you may also have to update the [Opaque/Bytestring](https://reference.opcfoundation.org/v105/Core/DataTypes/ByteString/) tags. Consider using /scripts/migration/update_opaque_tags.py and modifying the script to fix the issue. If that doesn’t work, then you’ll have to go through each tag names and find the identifier value manually.

v2.20.0 (2022-10-19)

Features

  • Added the ‘open_weather_api’ receiver

  • Added support for kusto on the sql receiver. (Instead of KQL, it is recommended to continue using SQL for kusto)

v2.19.0 (2022-09-26)

Highlights

  • Removed ‘kafka’ receiver and ‘kafkaproducer’ transmit. Aside from not being used, removing these will speed

up development time when building docker images. * Upgraded SQLAlchemy to 1.4.41 * On the sql receiver, ‘isolation_level’ is now set to AUTOCOMMIT which means explicit transactions are turned off.

v2.18.1 (2022-09-12)

Bug Fixes

  • Fixed a bug with Enact receiver where it crashes when data is empty.

  • Fixed a bug with Enact receiver where state is being deleted due to misnaming.

Upgrade Notes

  • If already in use, enact receiver will need to be restreamed.

v2.18.0 (2022-09-09)

Features

  • Implemented Enact InfinityQS receiver

v2.17.0 (2022-08-31)

Features

  • Added OPC-UA support for nested lists on polling and pubsub subscription

Bug Fixes

  • Fixed a bug with JSONSchema where default values would be modified.

  • Added validation errors to OPC-UA receiver to distinguish polling and subscription settings.

  • Remove excessive logs from piweb receiver.

v2.16.1 (2022-07-20)

Features

  • The SQL receiver has a new option called max_connection_wait. It controls the number of seconds to wait to acquire a connection. This can be helpful for receivers with a large number of streams that run into problems with QueuePool connection timeouts.

v2.16.0 (2022-07-14)

Features

  • Add support for reading from OPC-UA component nodes in events.

  • Add support for reading OPC-UA event nested fields.

v2.15.1 (2022-05-27)

Bug Fixes

  • Add commands that were lost in 2.14.0. (ping, nslookup, dig)

v2.15.0 (2022-05-20)

Features

  • Added a piweb receiver. (Named piweb_v1 in the config)

  • Added support for ByteString, ByteStringHex identifier types added in OPC-UA. (This is part of an ongoing effort to add a historical read receiver for OPC-UA)

  • sight_machine_cloud included as a JSON section link on the configuration screen.

Bug Fixes

  • Fixed an issue where only the first receiver shows the SQL pop-up editor

  • Fixed an issue where Ctrl+F searches would trigger the SQL pop-up editor

v2.14.0 (2022-03-31)

Highlights

  • Upgrade debian to v11 (bullseye)

    • Python upgraded from 3.7.2 to 3.7.12

    • FreeTDS upgraded from 1.3.1 to 1.3.4

    • …and other minor changes.

Bug Fixes

  • Fixed an issue where untagged customer environments would fail during CircleCI tests.

v2.13.1 (2022-03-14)

Bug Fixes

  • Fixed an issue where where the OPC-UA pubsub subscription fails to reconnect after the server shuts down.

v2.13.0 (2022-03-07)

Highlights

  • Added ‘identifier_type’ and ‘default_identifier_type’ parameters to allow support for the following Node ID types: String, Numeric, and Guid when using OPCUA pubsub subscription or polling. Any config without these parameters will default to the String type.

v2.12.1 (2022-02-17)

Bug Fixes

  • Fixed bug where OPCUA event subscription buffer was not being emptied so all messages would be duplicated.

v2.12.0 (2022-02-15)

Features

  • Added implementation for the opcua event model which means we now have two subscription models: pubsub and event. Along with that, we’ve added a few additional options:

    • Each stream provides a subscription_type option. The same receiver can handle both pubsub and event subscriptions simultaneously.

    • server_node_id and node_id for configuring which events to subscribe to. They can be configured using either a Node ID or BrowseName notation.

Bug Fixes

  • Updated opcua 0.98.13 -> 0.98.13+sm1. This patch solves an issue where custom structures stored in the OPC Binary Schema could not be loaded for the opcua event model.

v2.11.0 (2022-02-07)

Highlights

  • Upgrade opcua 0.98.5 -> 0.98.13. The newer version supports Basic256Sha256.

v2.10.0 (2021-12-14)

Highlights

  • Include timestamp as a payload field in rdpv3.

  • Add 503 error message handling to show kafka errors in rdpv3.

Bug Fixes

  • Added constraints.txt to integration-tests to avoid test failure.

v2.9.1 (2021-09-01)

Bug Fixes

  • The SQL receiver now respects the max_concurrent_connections setting. Previously each receiver could create up to 10 additional concurrent connections, each of which would be closed after each query.

    Closes FTX-699.

v2.9.0 (2021-08-24)

Features

  • Updated pymssql 2.1.4 -> 2.2.2 and FreeTDS from 0.91 -> 1.3.1, for SSL support in particular.

Bug Fixes

  • Fix ftx issue leading to confluent_kafka buffer filling up in kafkaproducer.

  • Fix broken validation ranges for max_request_size_bytes and max_request_records. Additionally, max_request_size_bytes exceeds 4 MB, the system will still run but it will be set to 4 MB.

v2.8.0 (2021-08-11)

Upgrade Notes

  • pyasn1 0.1.8 -> 0.4.8. This was blocking some packages used in factorytx-demo.

Features

Added a new transmit called rdpv3 (mostly built off of rdpv2).

  • Includes existing options of both rdpv2 and kafka (kafka ones have been renamed)

  • Data is transported to ma via JSON instead of BSON

  • The ma endpoint (not FTX) will upload data to kafka. And it will be in the form of raw data similar to the kafka transmit.

v2.7.2 (2021-07-21)

No highlights here. Just fixing a broken mypy-related test that leaked into 2.7.1.

v2.7.1 (2021-07-01)

Highlights

  • Supports the new, optional topic_field parameter in the data_transmit section of the FactoryTX configuration, for transmit_type kafkaproducer.

Features

When the optional topic_field option is present, the value of that option (the name of a field in the dataframe) is used to identify the topic suffix for each record. If the option is not present, then the default topic suffix (the stream ID) is used instead.

  • new_repository.sh should be creating an empty requirements.txt file.

v2.7.0 (2021-04-05)

Highlights

Upgrade Notes

  • Existing customer repositories will need to do the following:

    • Remove the following two lines from requirements.txt: .. code-block:

      --no-index
--find-links=https://sm-mirror.s3-us-west-2.amazonaws.com/pip/index.html

    • Update the current CircleCI config files by checking out this release of factorytx-core (or newer), deleting the existing config files, and running the setup_circleci_config.sh script to recreate it.

Features

  • Added oracle support to the SQL Data Receiver. (New configuration specific to that database type has been added: service_name, sid)

Bug Fixes

  • Remove unnecessary circleci build step of checking python package requirements.

  • Keep pip package links in factorytx-core and not copied over to customer repos.

v2.6.1 (2021-02-08)

Upgrade Notes

  • Existing customer repositories will need to update the current CircleCI config files by checking out this release of factorytx-core (or newer), deleting the existing config files, and running the setup_circleci_config.sh script to recreate it.

> cd /opt/sightmachine/factorytx-<customer>
> rm .circleci/config.yml
> (cd factorytx-core && git fetch && git checkout v2.6.1)
> ./factorytx-core/scripts/setup_circleci_config.sh <customer>

Bug Fixes

  • OPC UA connection errors are logged instead of killing the receiver process with an uncaught exception. Closes FTX-630.

  • Excel spreadsheets with blank first columns can now be re-parsed if they are updated. Previously the second parsing pass would cause a KeyError. Closes FTX-663.

  • All dependency versions are now pinned in constraints.txt. This fixes a bug where the wrong versions of some dependencies were sometimes installed. Closes FTX-664.

v2.6.0 (2020-10-30)

Highlights

  • All factorytx container builds pull from Google Cloud Registry (GCR) vs DockerHub due to rate limiting. Closes FTX-646

  • Upgraded tool usage from balena legacy resin-cli to new balena-cli. This requires all developers to upgrade as well. Closes FTX-647.

Upgrade Notes

  • Existing customer repositories will need to update the current CircleCI config files by checking out this release of factorytx-core (or newer), deleting the existing config files, and running the setup_circleci_config.sh script to recreate it.

> cd /opt/sightmachine/factorytx-<customer>
> rm .circleci/config.yml
> (cd factorytx-core && git fetch && git checkout v2.6.0)
> ./factorytx-core/scripts/setup_circleci_config.sh <customer>

  • Developers will need to install new balena-cli as a replacement for the resin-cli as outlined in Deployment README.

Features

  • Added a compression configuration option to the CSV parser. This option allows usage of compressed files as data sources. (#98), part of the changes for DEMOS-137.

Bug Fixes

  • FTX Streams UI: Fixed an issue when trying to perform a bulk restream of multiple selected data streams. Now, the restream request will be successfully processed. (#102), closes FTX-621.

v2.5.1 (2020-08-21)

Highlights

  • Enabled support for customer repositories to add Python packages. (#96), closes FTX-638.

Upgrade Notes

  • Existing customer repositories will need to update the current CircleCI config files by checking out this release of factorytx-core (or newer), deleting the existing config files, and running the setup_circleci_config.sh script to recreate it.

> cd /opt/sightmachine/factorytx-<customer>
> rm .circleci/config.yml
> (cd factorytx-core && git fetch && git checkout v2.5.1)
> ./factorytx-core/scripts/setup_circleci_config.sh <customer>

  • Existing customer repositories will also need to add a requirements.txt file to the repository. The file should start with:

--no-index
--find-links=https://sm-mirror.s3-us-west-2.amazonaws.com/pip/index.html
# Add custom Python dependencies below

Bug Fixes

  • Standardized the timestamp headers in messages produced to RawQ by the kafkaproducer transmit. Each message should have edge_arrival_timestamp and timestamp headers in ISO-8601 format with nanosecond precision. (#95), related to work for MA-6864.

v2.5.0 (2020-07-01)

Highlights

  • The kafka Receiver has been deprecated. It should only be used for testing and debugging purposes.

  • The max_request_size_bytes option of the Kafka and remotedatapost transmits has been deprecated. It should no longer be used. (#92), addresses FTX-627.

  • Running integration tests for FactoryTX customer repositories has been deprecated. The integration tests will be disabled by default for new customer repositories. Existing customer repositories will need to have integration tests disabled. Please refer to the “Upgrade Notes” section for more details. (#93), addresses FTX-629.

Upgrade Notes

  • Existing customer repositories will need to prune integration tests from current CircleCI config files by checking out this release of factorytx-core (or newer) and running the setup_circleci_config.sh script.

> cd /opt/sightmachine/factorytx-<customer>
> (cd factorytx-core && git fetch && git checkout v2.5.0)
> ./factorytx-core/scripts/setup_circleci_config.sh <customer>

Features

  • The kafkaproducer Transmit has a new configurable option: enable_idempotence, which allows for the use of Kafka’s Idempotent Producer for exactly-once producing. (#82), related to work completed to close MA-6672.

  • Added Matplotlib and tqdm to FactoryTX, so developers can leverage these tools in their Jupyter Notebooks. (#85), closes FTX-615.

  • Updated the kafkaproducer Transmit to require each record have a timestamp field configured. Each record’s event time will be sent to Kafka in the header as an ISO-8601 string and in the “timestamp” field of the Kafka message as an Unix epoch in milliseconds, which Kafka uses automatically for indexing. (#88), closes FTX-618.

Bug Fixes

  • Updated Azure Data Lake credentials used in an integration test and in the FactoryTX manual. (#86), closes FTX-617.

  • Updated error handling when trying to connect to and list files with the Azure Data Lake receiver. (#87).

  • receivers.file.receiver: Instead of logging an INFO-level message, the receiver will save an error marker if downstream operations are unable to process a file. The error marker will be cleared when the file is successfully processed. (#89), closes FTX-619.

  • Nginx: Updated the Nginx configuration to allow up to 5 MB in the body of an API request. (#90), addresses SU-1223.

  • Uncaught exception markers always resolve after the process or thread that created them has stayed up for ten minutes. (#91), addresses FTX-626.

v2.4.2 (2020-01-16)

Features

  • Kafka Transmit will use canonical JSON so ETL3 (Flinktory) can perform deduplication. (#78), part of the work for MA-6058.

Bug Fixes

  • Deprecated validation check for minimum length of username and password in the Kafka Transmit. (#77)

  • Updated expired certificates used in an OPC-UA integration test and performed some code cleanup. (#79), closes FTX-607.

v2.4.1 (2019-11-05)

Bug Fixes

  • transforms.checksum_record: Fixed error when trying to transform a DataFrame with more than 255 columns. (#74), closes FTX-594.

  • CircleCI: Disabled triggering builds when Git tags are created. (#75)

v2.4.0 (2019-10-24)

Highlights

  • Migrated from Jenkins to CircleCI for continuous integration (CI) support. Now, when a user pushes commits to the repository, CircleCI (instead of Jenkins) will handle building the FactoryTX image, uploading it to Sight Machine’s Google Cloud Registry, running unit and integration tests, and publishing the documentation.

  • Updated the README to include information about FactoryTX images and deploying to FactoryTX Cloud Kubernetes instances. Also, added a script to help make changes to FactoryTX Cloud Kubernetes instances.

  • Added a transform that can create a checksum value based on a record’s values.

Upgrade Notes

  • Jenkins to CircleCI Migration: Existing customer repositories will need to follow transition instructions detailed in the README. New customer repositories will automatically assume that CircleCI is the CI platform used.

  • FactoryTX Configuration: connection_name is now a required setting for file receiver connections. If you are updating existing receiver connections with a name and want to maintain state, you’ll need to use the naming convention that the FactoryTX API previously used for assigning default names. The name will start with Connection followed by an integer representing the connection’s position in the list. For example, the first connection would have had the default name Connection 1.

Features

  • Added process_files_alphabetically and process_ordered_connections options to file receiver configuration schemas, so the user can now define the order of processing received files. process_files_alphabetically enables processing files in alphabetical order based on their file name (includes path). process_ordered_connections enables processing files in groups based on the order of the configured connections (if there is more than one connection). Please refer to the configuration reference for more details on using these new configuration options. (#39), closes FTX-563.

  • Added a temporary_file_directory option to the localfile receiver, so the user can define where to temporarily store files downloaded from the localfile connection(s). (#41), closes DEVOPS-4743.

  • Added more information about the FactoryTX build pipeline, specifically about images. A script has been added to help with making changes to FactoryTX Cloud Kubernetes instances. (#43), closes FTX-571.

  • Upgraded Kafka libraries to support ssl.identification.algorithm. (#47 and #49), closes FTX-576 and FTX-578.

  • Added S3 connection options to retrieve security credentials with an AssumeRole API call. (#53), related to DATA-201.

  • Added build information to the FactoryTX image as part of the migration to using CircleCI workflows. The same logic has been applied to the Balena deployment script (#66).

  • Added validation logic in the UI to check for file receiver connection changes and throws a warning message in a browser window prompt. (#67), related to FTX-538.

  • Added a transform that can create a checksum based on a record’s values to help identify duplicate records. It should be noted that the checksum is a promoted field in the RDPv2 transmit, so the checksum is copied out of a record’s data.fieldvalues. (#69), closes FTX-589.

Bug Fixes

  • API.v1.status: To match how Kubernetes calculates available memory, we calculate the working set of memory and compare it to the total available memory. (#42), closes FTX-573.

  • transmits.kafka: Switched to using the Pandas to_json() method when converting a DataFrame into JSON to handle NaN values. (#45), closes FTX-574.

  • scripts

    • post-commit: Fixed an issue with compatibility of double brackets ([[ ]]) between sh and bash on Linux and Mac. (#48)

    • new_repository and deploy: Fixed a compatibility issue with the -i option for sed between Linux and Mac. (#51), closes FTX-581.

  • receivers.file.receiver

    • Improved cleaning up of temporary files when a file receiver fails to process a file, so we don’t unnecessarily fill up our temporary storage. (#56), closes FTX-579.

    • Don’t log connection info if receiver cannot connect. (#61)

  • README: Clarified where to launch FactoryTX when using factorytx-customer repositories. (#57)

  • FTX Streams UI: Fixed an error with “Select All” in the Streams UI. Now, when the user clicks “Select All”, only visible rows are selected. (#70), closes FTX-516.

  • CircleCI: Separated commit and pull request CircleCI workflows. (#71), closes FTX-587.

v2.3.1 (2019-06-24)

Highlights

  • Removed the tenant alias “orora_replicate” for the “replicate” transform. Please check and update your FactoryTX configuration accordingly.

  • Improved handling of data received from some of the OPC-UA tags in a subscription. Previously, we would restart the pub/sub connection; now, we will maintain the connection and process the data that was received.

  • Added an option to the local file receiver connection to control whether or not to recursively search for files in the data directory.

Features

  • Removed the tenant alias “orora_replicate” for the “replicate” transform. (#33), closes FTX-532.

  • Changed default compression option to “snappy” since the Flink Java Client doesn’t easily support “zstd” yet. (#37)

  • Added a recursive option to the localfile receiver connection. When enabled (default), the connection will recursively search for files in the specified data directory. If disabled, the connection will only look for files in the data directory. (#38), closes FTX-569.

Bug Fixes

  • API.test-data: Removed JSON files of deprecated API endpoints and their responses because the filenames were incompatible with Windows file systems. (#34), related to FTX-555.

  • receivers.opcua

    • Check for duplicate exportnames from OPC-UA tags within a stream. Allow for duplicates across streams.

    • When using the OPC-UA pub/sub model, if we reach the subscription timeout and only have data for some of the subscriptions, we will now mark a connection warning and process the data that we did receive. (#29), closes DATA-130.

    • When using the OPC-UA pub/sub model, the initial subscription data check takes about the same time as the poll_interval (or longer). We will now wait the full duration of the poll_interval after the initial subscription check to prevent duplicate data from getting read. Subsequent subscription checks will still wait based on the difference between the poll_interval and the amount of time it took to retrieve the data. (#29), related to tests done for DATA-96.

  • transmit.kafka: Fixed handling of NaN values in the transmit component. (#36), closes RAWQ-42.

v2.3.0 (2019-05-13)

Highlights

  • Better accuracy when reporting CPU and memory usages in FactoryTX containers deployed on Kubernetes.

Features

  • Added a script that recursively removes __pycache__ subdirectories. (#24), closes FTX-527.

Bug Fixes

  • API.v1.status: We will attempt to calculate CPU and memory usages in a container based on values read from the cgroup filesystem, which should provide a more accurate representation of the container’s resource consumption. If the cgroup filesystem isn’t mounted or cannot be read, we fallback on reading values from the /proc directory. (#23), closes FTX-509.

  • .dockerignore: Updated to ignore __pycache__ folders and .pyc, .pyo, and .pyd files in subdirectories when building Docker images. (#24)

  • receivers.kafka.receiver._KafkaStream: Kafka consumers kept timing out after a few minutes. To workaround the issue, we recreate the consumer after five zero-message responses from the Kafka server. (#27 <https://github.com/sightmachine/factorytx-core/pull/27>), closes DEVOPS-4587.

  • Documentation: Updated documentation references of the factorytx2 repository to factorytx-core (and/or factorytx-<customer>). (#31), closes FTX-546

v2.2.0 (2019-04-30)

Highlights

  • Updated the Debian base image from Jessie to Stretch, and Python from 3.6 to 3.7.2. A few dependencies and Python packages were added and updated in the process.

  • Fixed issue where candidate reconciliation for File Receivers (LocalFileReceiver & S3Receiver) went into a spinlock due to a poor table join.

Upgrade Notes

  • DB Migrator: The migrations for the State DB have been moved to the StateDBMigrator class so then can be extended on a per receiver basis. The first such receiver to customize their DB migrations is the FileReceiver. Any future migrations should follow a similar pattern to what the FileReceiver does.

  • Named Connection Migration: The FileReceiver (LocalFileReceiver & S3Receiver) now has a migration which forces all entries in the State DB for the Receiver to contain a named connection. If the migration script is unable to reconcile an entry/row in the State DB with a file on the Receiver, the entry will be deleted and a log message emitted detailing the content deleted.

Features

  • Updated the base image to use Debian Stretch and Python 3.7.2. (#18), closes FTX-526 and FTX-530.

Bug Fixes

  • Fixed issue where candidate reconciliation for File Receivers (LocalFileReceiver & S3Receiver) went into a spinlock due to a poor table join. (#25), closes FTX-537.

v2.1.0 (2019-04-10)

Highlights

  • RawQ support: NEW FactoryTX receiver (Kafka Data Receiver) and transmit (Kafka Transmit) for receiving and transmitting data to an Apache Kafka server. Please refer to the Advanced Topics - Kafka article in the Concepts section of the FactoryTX Manual for more information about Kafka and the RawQ project.

  • Added a tool for visualizing how FactoryTX configurations apply to streams.

  • Support for Prometheus metrics.

  • Added support for “start after” polling for file receiver connections, which retrieves files based on the last completed file. Currently, only the S3 file receiver connection takes advantage of the new feature with the retrieve_ordered_files configuration setting.

Upgrade Notes

  • Custom file receivers: A start_after_hint parameter has been added to the base file receiver’s list method. Although the factorytx.receivers.file.base.Connection.list() method signature has changed, we will temporarily support Connection.list implementations without any parameters. To avoid problems with future upgrades, please add the start_after_hint parameter to the list method of your custom file receiver implementations.

Features

  • Added support for overlapping paths in multiple file receiver connections. (#17), closes FTX-525.

  • Added support for converting epoch time values into a datetime value in the convert_timestamps Transform. (#13), related in closing WRIM-174.

  • The UI will now display a confirmation window for overwriting the saved configuration when data receiver names have been changed. (#14), closes FTX-518.

  • Added support for “start_after” polling for file receiver connections. The first file receiver connection to support this feature is the s3 connection with the retrieve_ordered_files configuration setting (#11), closes FTX-521.

  • Added tool for visualizing how FactoryTX configurations apply to streams (#10), closes FTX-513.

  • Added support for receiving and transmitting data to an Apache Kafka server (#9), related to DEVOPS-4111.

  • Added a Prometheus Client integration to track events over time (#7), closes FTX-524.

  • Validate that the Python environment does not contain packages with conflicting or missing requirements (#5), closes FTX-522.

Bug Fixes

  • receivers.sql

    • Improved the error message when the SQL receiver encounters a null value in a state field (#21), closes FTX-409.

    • Updated the SQLAlchemy Python library to v1.2.7 to take advantage of the pool_pre_ping option. The receiver will perform a liveness test upon checking out a SQL connection (#20).

  • transforms.convert_timestamps: Allow for integer-based indexing on a pandas.Series, resolving an error message thrown during DST (#8), closes FTX-523.

  • README: Updated the factorytx-core README with FactoryTX split instructions (#19), closes FTX-502.

  • ftxui: Use relative paths for “ftxui” when using the watch script (#6)

v2.0.1 (2019-03-12)

Highlights

  • FactoryTX-core Split: FactoryTX is now separated into two types of repositories:

    1. factorytx-core: Core FactoryTX components and functionality.

    2. factorytx-customer: Custom components and functionality for a customer.

Breaking Change

The factorytx2 repository has been deprecated. Core components and functionality has been migrated into the factorytx-core repository.

Customer-specific components are now stored in GitHub repositories separate from core components and functionality. If a factorytx-customer repository has not been created yet, please follow the instructions in the Creating a new customer repository section of the FactoryTX Split document.

A factorytx-customer repository uses factorytx-core as a Git Submodule. Please note the process flow changes documented in the FactoryTX Split document.