Efficient Apartment Hunting with FME Automation: A Step-by-Step Guide

Setting Up Email Alerts

Triggers in FME Flow

Each time a new apartment listing hits the market, an email notification is promptly dispatched to our trusty lizard companion. A fast integration and comparison, thanks to this email alert, will put us in a better position to make an offer when time is of the essence!

In FME Flow, adding this automated alert is as simple as a drag and drop action. Here’s how to set it up:

1. Load the configuration for your email provider, such as Gmail.
2. Facilitate testing by adding Manual trigger with a shared custom attribute.

Every time a new apartment comes to the market, an email is sent to our lizard. This is called a Trigger. In FME Flow you just need to drag and drop to add one.

Set up the Email trigger

1. Add a trigger object and select the trigger type.
2. Load the configuration for your email provider, such as Gmail.
3. Enter your email credentials for seamless integration.

Add a trigger object

Select the trigger type

Load and configure your email account

• Check the result

To ensure everything is running smoothly, add a log message action after setting up the trigger. This will provide a visual representation of the email content, allowing for quick verification of data retrieval.

Manual trigger for testing purposes

Additionally, consider adding a manual trigger for testing purposes. This feature enables you to input a default or test value, ideal for validating the automation process. Given the current state of the Finnish real estate market, this manual trigger proves invaluable for testing the automation without relying solely on a constant influx of new listings.

1. Add a trigger object and select the Manual Trigger type.
2. Set a value which will be pushed as a message when the trigger is pushed. It should be what you got from an email.content from the other trigger.
3. Create a custom attribute on both triggers, here “user.emailcontent”, containing either email.body or “manual” content.

Output available out of an email trigger, with custom attribute configuration

• Test it !

With these steps in place, testing the automation is a breeze. Click “Start” and then “Trigger” to initiate the process. Voila! Your first automation is up and running. While the current output may be a simple text format with address, price, and a link in an unconventional encoding, fear not—we’ll refine this with FME Form in the next steps.

Part of the running Automation interface

Etuovi.com 1 uusi kohde löytynyt! Hei WT2020, Vahtisi Tampere on löytänyt 1 uuden tai hinnaltaan muuttuneen kohteen, joka vastaa hakuehtojasi. streetname 5 B, 33720, area, Tampere, Suomi https://www.etuovi.com/kohde/1111111

Data Extraction, Web Scraping and Geocoding for Additional Details

In this phase, our goal is to transform the raw email message into structured data containing the apartment characteristics, ready for upload to our spreadsheet.

Data quality and preparation are essential for building a strong and reliable data pipeline. Therefore, we’ll take the time to delve into these crucial steps. If you’re not inclined to explore these details, feel free to skip ahead.

As this requires more intricate processing, we’ll utilize FME Form to handle these complex steps before seamlessly integrating the pipeline back into FME Flow.

FME Form Steps

1. Extract Information from the Email Message
2. Enhance with Web Scraping
3. Geocode the Address
4. Integrating with FME Flow Automation

• Extract Information from the Email Message

FME Flow pushes the email message through a parameter, requiring definition and retrieval using ParameterFetcher.

Navigate encoding complexities by employing a combination of AttributeEncoder transformations.

Leverage StringSearcher and Regex to efficiently extract address components and webpage links, organizing them into a structured list.

Finalize data preparation using AttributeManager.

End result

• Enhance with Web Scraping

Here we want to extract as much as we can without making the process to rigid. Tackling modern webpages’ complexities requires strategic tools. Here are some tools worth learning.

HttpCaller to retrieve page content.

HTMLExtractor for targeted text extraction via CSS selectors.

StringReplacer for text manipulation.

Here is the result of the extraction from the price section shown before.

• Geocode the Address

Real estate’s golden rule emphasizes location. With FME’s dedicated transformer, effortlessly geocode addresses to unlock spatial insights crucial for decision-making.

• Prepare the message to be pushed back to the Automation

Transitioning from FME Form back to FME Flow is seamless. By mapping attributes, effortlessly integrate the refined dataset into the automation pipeline.

Automation Writer Feature Type

FME Flow Steps: Deploying the Workspace and Adding to the Automation

Once your workspace is ready for deployment in FME Flow, the process is remarkably straightforward:

1. Publishing the Workspace: to make it accessible for the Automation
2. Adding it to the Automation: to link to the previous steps
3. Testing the Deployment

• Publishing the Workspace

Begin by clicking on the “publish” option. This initiates a guided process where you simply need to follow the prompts, clicking through “next, next, next, done.” This intuitive flow ensures a hassle-free publishing experience, even for those new to the platform.

• Adding it to the Automation

After publishing, integrate the workspace into your automation pipeline. This step involves adding an action within FME Flow. Since running a workspace functions as an action, simply select your workspace from the available options. Then, feed it with the relevant message data, conveniently stored in the custom attribute.

• Testing the Deployment

Before finalizing the integration, it’s essential to conduct thorough testing. This ensures that the workspace functions as intended within the automation pipeline. With a simple drag drop, add an action to log the message. Then initiate the test by starting the Manual Trigger.

Pushing Data to Google Sheets

While it may seem tempting to utilize FME Form directly for pushing data, doing so could inadvertently limit future flexibility. By incorporating the data push to Google Sheets within FME Flow, you ensure a more adaptable pipeline. Should your automation’s requirements evolve—perhaps necessitating data storage in an Excel file in S3 or a database—the flexibility of FME Flow allows for adjustments.

FME Form steps

In the FME Form process, several key components ensure efficient data handling:

1. Generating Feature IDs: this ensures robust data management and tracking capabilities.
2. Extracting Automation Messages: this converts the relevant automation messages to attributes.
3. Pushing to Google Sheets: this relies on Google Sheet dedicated writer.
4. Pushing Back to Automation: it opens the door for other potential steps on the automation. Pushing the UUID back is critical to maintain data integrity.

FME Flow steps

1. Publish the Workspace
2. Add it as an Action
3. Link Attributes and Parameters

Result

With the data successfully pushed to Google Sheets! This streamlined automation process not only simplifies apartment hunting but also demonstrates the power of FME in creating flexible and efficient data pipelines.

Conclusion

With just two well-crafted workspaces and a few simple actions in FME Flow, we’ve established a seamless process for monitoring email alerts, extracting information from webpages, and pushing refined data to our cloud-based spreadsheet—all without any coding in Python, JavaScript, or reliance on external APIs.

FME’s strength lies in its flexibility to integrate with various external services seamlessly, enabling organizations to break down data silos and enhance efficiency and collaboration. Transitioning to web-based automation and apps empowers non-coders to design tailored workflows, fostering innovation and driving success in the digital age.