You are viewing documentation for an older version.

Click here to view the latest documentation.

Profiles 0.3.x Changelog

Version 0.3.1

3 February 2023

This version addresses a crucial defect, so please make sure to update your version. Note that you won’t have to update your schema for this release.

Version 0.3.0

25 January 2023

We have got a new name! WHT is now called Profile Builder (PB), RS360 is now Profiles. Be sure to check out our newest release that comes with several new features for an enhanced user experience.

What’s New

• Migrate - A new command that will enable you to migrate your project to a newer schema. It has two subcommands:

  • Manual - You will get to know steps you need to follow to manually migrate the project yourself. It will include both breaking and non-breaking changes.
  • Auto - Automatically migrate from one version to another.

• We have made a few significant changes to YAML. The changes consist of:

  • Bumping schema version from 9 → 18.
  • Entityvar (Feature Table) - We have renamed tablevar, tablefeature and feature to entityvar; as they all were adding columns to an entity with nearly identical YAML. A new vars: section of feature table YAML contains list of inputvars and entityvars. Whereas features: field same YAML is a list of entityvar names which should be part of the final output table.
  • ID Stitcher is now linked to an entity. As a result, all tables using that entity will use the linked ID Stitcher. Earlier, an ID stitcher was linked to a feature model.
  • Some of the terms in yaml spec are changed to make it closer to SQL terminologies. For entityvar and inputvar spec: value → select, filter → where , ref → from. In inputs spec: sql → select.
  • Project file has a new key named include_untimed. If set to true, data without timestamps are included when running models. This reduces data errors for timestamp materials. Also, we have deprecated the flag require-time-clean in the run command.
  • Id types can now be re-used between entities. In the project file, entities now have a list of id types names, instead of a list of definitions. In the inputs file, a required entity field is added to the ID list that specifies which entity this ID type is being extracted for.
  • Now an inputvar can also read from a macro, just like tablevar.
  • Global Macros - You can now define macros in a separate YAML file inside your models folder. They can then be used across different models in your project. Thus a macro becomes independent that can be reused across multiple models.
  • wht_project.yaml is renamed to pb_project.yaml and ml-features.yaml to profiles.yaml.

• Cleanup Materials - A new command that allows you to review all the created materials and then delete them (NOTE: experimental feature).

• Discover - A new subcommand discover materials has been added. Using it, you can now discover all the materials associated with a project.

• Compile/Run - GIT URL now supports tags. To use, execute the command pb compile -w git@github.com:<orgname>/<repo>/tag/<tag_version>/<folderpath>.

Improvements

• Web app - The UI is now more intuitive and user-friendly.
• Log tracing is now enabled by default for most commands. Log files are stored in logs/logfile.log of your current working directory. They store upto 10 MB data. Also, the logger file now stores more granular information for easier debugging in case of unexpected errors.
• Significant performance improvements in creating ID stitched tables, in case a lot of duplicates are present.
• Add extra columns (Hash, SeqNos) to differentiate between entries for commands to discover sources and entities.
• When you execute a profile via run command, then the generated SQL gets saved in the output folder.
• Added .gitignore file to init project command, to prevent unnecessary files being added to GIT Repo. Such as, .DS_Store, output and logs folders.
• Tonnes of changes under the hood.

Bug Fixes

• Fixed the bug where window functions were creating multiple rows (duplicates) per main id.
• Resolved the bug in inputvars which was doing joins on main_id instead of row id.
• Executing the command init profile now inputs values in the same order as on the web app.
• Resolved the bug where extra gitcreds[] and warehouse lines were added on overwriting a profile that already existed.
• A few redundant parameters were being shown in the validate access command which have been removed.
• Removed a couple of redundant subcommands in the init project.

Known Issues:

• Warning: While the run command is being executed, canceling it by pressing Ctrl+C doesn’t work as expected. Though it will stop the program’s execution on the CLI, the query will keep running on the data warehouse. This is a documented Snowflake behavior.
• In a model, an input can’t use columns named “MAIN_ID”, “OTHER_ID”, “OTHER_ID_TYPE”, or “VALID_AT” in its ID SQL.
• When creating a profile via init command, pressing the Ctrl+C command doesn’t exit the application.
• Web app doesn’t allow you to select a date older than 30 days.
• Migrate auto jumbles up the order and removes comments.

Questions? Contact us by email or on Slack