Efficient Apartment Hunting with FME Automation: A Step-by-Step Guide
Introduction
Searching for the perfect apartment can be time-consuming and boring, but with the power of automation, we can simplify the process. In this tutorial, we’ll walk through how to set up an automated workflow using FME (Feature Manipulation Engine) to streamline the apartment search process of our FME Lizard in Tampere, Finland. Let’s dive in!
Steps
Setting Up Email Alerts
Data Extraction, Web Scraping and Geocoding for Additional Details
Pushing Data to Google Sheets
Setting Up Email Alerts
Triggers in FME Flow
Each time a new apartment listing hits the market, an email notification is promptly dispatched to our trusty lizard companion. A fast integration and comparison, thanks to this email alert, will put us in a better position to make an offer when time is of the essence!
In FME Flow, adding this automated alert is as simple as a drag and drop action. Here’s how to set it up:
Load the configuration for your email provider, such as Gmail.
Facilitate testing by adding Manual trigger with a shared custom attribute.
Every time a new apartment comes to the market, an email is sent to our lizard. This is called a Trigger. In FME Flow you just need to drag and drop to add one.
Set up the Email trigger
Add a trigger object and select the trigger type.
Load the configuration for your email provider, such as Gmail.
Enter your email credentials for seamless integration.
Add a trigger object
Select the trigger type
Load and configure your email account
Check the result
To ensure everything is running smoothly, add a log message action after setting up the trigger. This will provide a visual representation of the email content, allowing for quick verification of data retrieval.
Manual trigger for testing purposes
Additionally, consider adding a manual trigger for testing purposes. This feature enables you to input a default or test value, ideal for validating the automation process. Given the current state of the Finnish real estate market, this manual trigger proves invaluable for testing the automation without relying solely on a constant influx of new listings.
Add a trigger object and select the Manual Trigger type.
Set a value which will be pushed as a message when the trigger is pushed. It should be what you got from an email.content from the other trigger.
Create a custom attribute on both triggers, here “user.emailcontent”, containing either email.body or “manual” content.
Output available out of an email trigger, with custom attribute configuration
Test it !
With these steps in place, testing the automation is a breeze. Click “Start” and then “Trigger” to initiate the process. Voila! Your first automation is up and running. While the current output may be a simple text format with address, price, and a link in an unconventional encoding, fear not—we’ll refine this with FME Form in the next steps.
Part of the running Automation interface
Etuovi.com 1 uusi kohde löytynyt! Hei WT2020, Vahtisi Tampere on löytänyt 1 uuden tai hinnaltaan muuttuneen kohteen, joka vastaa hakuehtojasi. streetname 5 B, 33720, area, Tampere, Suomi https://www.etuovi.com/kohde/1111111
Data Extraction, Web Scraping and Geocoding for Additional Details
In this phase, our goal is to transform the raw email message into structured data containing the apartment characteristics, ready for upload to our spreadsheet.
Data quality and preparation are essential for building a strong and reliable data pipeline. Therefore, we’ll take the time to delve into these crucial steps. If you’re not inclined to explore these details, feel free to skip ahead.
As this requires more intricate processing, we’ll utilize FME Form to handle these complex steps before seamlessly integrating the pipeline back into FME Flow.
FME Form Steps
Extract Information from the Email Message
Enhance with Web Scraping
Geocode the Address
Integrating with FME Flow Automation
Extract Information from the Email Message
FME Flow pushes the email message through a parameter, requiring definition and retrieval using ParameterFetcher.
Navigate encoding complexities by employing a combination of AttributeEncoder transformations.
Leverage StringSearcher and Regex to efficiently extract address components and webpage links, organizing them into a structured list.
Finalize data preparation using AttributeManager.
End result
Enhance with Web Scraping
Here we want to extract as much as we can without making the process to rigid. Tackling modern webpages’ complexities requires strategic tools. Here are some tools worth learning.
Web scrapping steps
HttpCaller to retrieve page content.
HTMLExtractor for targeted text extraction via CSS selectors.
StringReplacer for text manipulation.
Here is the result of the extraction from the price section shown before.
Geocode the Address
Real estate’s golden rule emphasizes location. With FME’s dedicated transformer, effortlessly geocode addresses to unlock spatial insights crucial for decision-making.
Prepare the message to be pushed back to the Automation
Transitioning from FME Form back to FME Flow is seamless. By mapping attributes, effortlessly integrate the refined dataset into the automation pipeline.
Automation Writer Feature Type
FME Flow Steps: Deploying the Workspace and Adding to the Automation
Once your workspace is ready for deployment in FME Flow, the process is remarkably straightforward:
Publishing the Workspace: to make it accessible for the Automation
Adding it to the Automation: to link to the previous steps
Testing the Deployment
Publishing the Workspace
Begin by clicking on the “publish” option. This initiates a guided process where you simply need to follow the prompts, clicking through “next, next, next, done.” This intuitive flow ensures a hassle-free publishing experience, even for those new to the platform.
First step when you click to publish a workspace
Adding it to the Automation
After publishing, integrate the workspace into your automation pipeline. This step involves adding an action within FME Flow. Since running a workspace functions as an action, simply select your workspace from the available options. Then, feed it with the relevant message data, conveniently stored in the custom attribute.
Drag and drop an action step
Workspace configuration in Flow
Testing the Deployment
Before finalizing the integration, it’s essential to conduct thorough testing. This ensures that the workspace functions as intended within the automation pipeline. With a simple drag drop, add an action to log the message. Then initiate the test by starting the Manual Trigger.
Pushing Data to Google Sheets
While it may seem tempting to utilize FME Form directly for pushing data, doing so could inadvertently limit future flexibility. By incorporating the data push to Google Sheets within FME Flow, you ensure a more adaptable pipeline. Should your automation’s requirements evolve—perhaps necessitating data storage in an Excel file in S3 or a database—the flexibility of FME Flow allows for adjustments.
FME Form steps
FME Form pipeline
In the FME Form process, several key components ensure efficient data handling:
Generating Feature IDs: this ensures robust data management and tracking capabilities.
Extracting Automation Messages: this converts the relevant automation messages to attributes.
Pushing to Google Sheets: this relies on Google Sheet dedicated writer.
Pushing Back to Automation: it opens the door for other potential steps on the automation. Pushing the UUID back is critical to maintain data integrity.
FME Flow steps
Publish the Workspace
Add it as an Action
Link Attributes and Parameters
Result
Output in Google Sheet
With the data successfully pushed to Google Sheets! This streamlined automation process not only simplifies apartment hunting but also demonstrates the power of FME in creating flexible and efficient data pipelines.
Conclusion
With just two well-crafted workspaces and a few simple actions in FME Flow, we’ve established a seamless process for monitoring email alerts, extracting information from webpages, and pushing refined data to our cloud-based spreadsheet—all without any coding in Python, JavaScript, or reliance on external APIs.
FME’s strength lies in its flexibility to integrate with various external services seamlessly, enabling organizations to break down data silos and enhance efficiency and collaboration. Transitioning to web-based automation and apps empowers non-coders to design tailored workflows, fostering innovation and driving success in the digital age.