
Flatfoot, pt 2 - Phoenix 1.3
Flatfoot
Note on this entry
This is an updated version of an earlier post, where I used the first release candidate for Phoenix 1.3 for my project. The released Phoenix 1.3.0 differed slightly from the release candidate, notably in where the web
directory is located. Those changes are all reflected here.
Capstone
The idea for my capstone project came from my wife. After one of those senseless, cyberbullying induced suicides, my wife though of an app where parents could monitor their loved one’s social media public feeds for any signs of bullying or outward displays of suicide ideation. In order to limit the scope of what promised to be a significant undertaking for a lone developer, I chose to collect on a single social media site initially. Given its robust API and general ease of use, I chose Twitter.
The Stack
For my tech stack, I chose to deviate from other languages I had already used. There wasn’t anything particularly wrong with Rails, JQuery, or AngularJS, but with an open sandbox and my pick of tools, I selected my favorites: Eilxir on the backend, with Phoenix providing the web interface layer, and React with Redux for the frontend demo. While Python or any number of other languages might have been more performant in terms of parsing and analyzing the collected data, few can compete with Elixir’s out of the box scalability, fault tolerance, and maintainability.
In order to develop a wider variety of skills in an academic setting, I chose to expose two different Phoenix endpoints as my user interaction layer. The first is a JSON API for handling all things related to user profile, settings, and authentication. For the other, I would use a famously efficient Phoenix Channel to manage websocket connections that expose the social media monitoring functionality of the app to the user. In regards to data collection from external API’s and analysis, I chose to utilize Elixir’s built in Erlang OTP abstraction.
Living on the Edge
Since this is an app I built in order to further my education, I opted to venture out on the bleeding edge and use Phoenix 1.3. Don’t let the what seems like a minor version change fool you. While there are no real breaking changes or significant new features being introduced in 1.3, alterations to the directory structure and new code generators introduce a seismic shift in how a developer approaches a Phoenix project. Chris McCord provides an overview of these changes in the Lonestar Elixir keynote address, his release candidate Elixir Forum post, version 1.3 forum post, and his more recent ElixirConf EU keynote. Additionally, astonj has compiled and maintains a Phoenix Contexts - learning resources. Those were all essential for me to getting started, but I found the concepts easier to grasp once I began building my project.
While the Phoenix team does emphasize that these changes are suggestions for how to organize and approach your project, for the purposes of my app I chose to implement it (mostly) as suggested. For me, these are the three big takeaways from this new version:
Web Location
- While Phoenix has only ever been the web framework for your Elixir app, the new directory structure makes that abundantly clear and better defines its role. Everything relating to collecting and serving Phoenix web endpoints is neatly packed away within the
lib/my_app_web
directory.
Contexts
- On a similar note, models are gone. Whereas Phoenix’s web interface was once built around models within the
/web
directory, we now extract that data fetching and manipulation (along with their implementation and support modules) to their own, self-contained systems. The functionality within those systems is exposed via contexts, which are dedicated modules that expose and group related functionality for an underlying system (i.e.clients.ex
is the context module for theClients
system and is located in thelib/my_app/clients
directory). These contexts allow us to decouple and isolate the systems from the remainder of the application and manage them independently. If that’s a bit confusing, just hang on - it’ll make mores sense as we get into the app.
Assets Location
- Assets no longer clog up the root directory. All of your external assets are stored in the
/assets
directory. Since I was using Webpack for this project, this was a particular pain for me initially, but ultimately it makes the project easier to navigate. If you’re interested, here’s how I setup and configured my Phoenix 1.3 app to work with Webpack 2, React, Redux, and React Router.
At its core, Phoenix 1.3 provides a forcing mechanism to design with intent. You have to think about what you’re going to do before you do it. Early on, I struggled a bit with the new structure, but ultimately these guidelines lead to more purposeful app development, with clear divisions of responsibility, and more reliable code.
The stuff dreams are made of
A quick note on the nomenclature for this project. In keeping with the Phoenix mythical bird theme, these systems are named after characters from the classic 1941 film, The Maltese Falcon. The relevant part of the movie’s premise is rather simple: A prospective Client approaches Bogart’s famous character, Sam Spade, with an interesting case. Spade takes the case and dispatches his partner, Miles Archer, to gather information on the case and report back, but ultimately it’s up to Spade to solve the mystery. Fortunately, we made a few improvements on the movie, so if Archer would happen to die under mysterious circumstances, he’ll be seamlessly revived and put back into action thanks to the magic of OTP.
I settled on the name Flatfoot for the app. I realize that’s more a beat cop than a private investigator like Spade, but 1940’s era slang for a PI doesn’t seem appropriate in a modern context.
Organizing in Phoenix 1.3
We can see a basic outline of the architecture quite clearly in the new directory structure. Each system has it’s own directory within /lib/flatfoot
. Also worth noting, the old /web
directory - once a clinger to the root in early versions of Phoenix - now finds a more appropriate home and a new name: /lib/flatfoot_web
.
Figure 1. Flatfoot directory tree in Phoenix 1.3
If you’re used to running a third party bundler like webpack with earlier versions of Phoenix, you’re also likely to notice how uncluttered the root directory is. Our package.json
, webpack.config.js
, .babelrc
, testing configs, and the node_module
directory are now all packed away quite neatly within /assets
.
We’re left with a clean, easy to reference directory tree, with clearly defined boundaries for our systems.
System design
Figure 2. Flatfoot systems overview.
In this app’s most basic form, all user interaction is managed by the FlatfootWeb
system. It’s responsible for handling requests from the user, requesting and persisting data via interaction with the other systems, and returning gathered data to the user. The Clients
and Spade
systems expose their interface to the Web
system via their context modules. The Clients
system handles all requests relating to the management of users and their preferences, such as creating a profile, editing that profile, and authorizing access via session tokens. The Spade
system is where the significant action begins.
Figure 3. Data flow when requesting new results.
Via websocket transport monitored by SpadeChannel
, users may request retrieval or deletion of persisted results or request new results altogether. Figure 3 depicts what happens when a user requests new results.
- On order,
SpadeChannel
will make a call to theSpadeInspector
context module, which in turn sends a request toSpadeInspector.Server
- an OTP module within the system. - That
Server
buildsconfigs
(which includes it’s ownpid
as a return address) for each social media account being monitored and sends thoseconfigs
to theArcher
context, which in turn sends those configs to the internalArcher.Server
module. - That
Server
then calls onArcher.FidoSupervisor
- aTaskSupervisor
- to initialize and monitor the backends (in this case justArcher.Backends.Twitter
). - Each backend will send results back to
SpadeInspector.Server
, per the return addresspid
. - Upon receipt of new results,
SpadeInspector.Server
system will evaluate those results, store them, and asynchronously return them to the user.
Contexts as API for App Boundaries
As Mikel Myskala has pointed out, the biggest mind shift here is understanding that we’re only minimally coupling our data to our application. One of the more challenging aspects was developing optimal boundaries for my application - and I’m still pretty sure I could further improve things.
Figure 4. Flatfoot’s boundaries
Unlike previous versions of Phoenix, accessing the underlying schema and modules should only be accomplished by calling functions within the context module - so no more Repo
calls from controllers. Within these contexts you’ll find common, RESTful functions like Clients.create_user(attrs)
or Clients.get_user!(123)
. But you can quickly create more dynamic and useful functions like Clients.get_user_by_token(token)
, which returns a User
when provided a valid session token. The Flatfoot.Clients
context functions as the boundary and it’s the only place you should expose the underlying system to the rest of the app.
Internal directory structures
Each system has it’s own directory, with a matching name context file. To aide in my own readability, I broke slightly with the out of the box Phoenix 1.3 directory convention. In order to avoid cluttering each system’s root directory with schema, I nested all of the Ecto
schemas within the /schema
directory:
|-- spade |-- schema | |-- backend.ex | |-- suspect.ex | |-- suspect_account.ex | |-- user.ex | |-- ward.ex | |-- ward_account.ex | |-- ward_result.ex | |-- watchlist.ex | |-- spade.ex <-- Click to view
It’s important to note that these systems can be more than just a single context file and schema. Any Elixir modules affecting the system can - and most likely should - be located within the boundary. In my Archer
system (see directory tree below), I included an archer/otp
directory to house my ArcherSupervisor
, Archer.Server
, and /backends
, which contains my task modules.
|-- archer |-- otp | |-- backends | | |-- twitter.ex | | | |-- archer_supervisor.ex | |-- server.ex | |-- schema | |-- backend.ex | |-- archer.ex <-- Click to view
The key to maintaining the boundary here, is that any interactivity with the server or backend schema goes through the archer.ex
context. For example, Flatfoot.Archer.fetch_data/1
exposes the Flatfoot.Archer.Server.fetch_data/1
function:
defmodule Flatfoot.Archer do
...
##########
# Server #
##########
alias Flatfoot.Archer.Server
def fetch_data(configs) do
Server.fetch_data(configs)
end
...
end
While this can appear unnecessary, this loose coupling provides a layer of isolation that let’s us modify our code internal to the system without having to consider that we are breaking any dependencies elsewhere in our app. A (nonsensical) example here, would be that if we wanted, we could change the name of the function from Server.fetch_data/1
to Server.get_me_some_data/1
. Instead of having to change every call to Server.fetch_data/1
, we only have to adjust it in the context:
...
def fetch_data(configs) do
Server.get_me_some_data(configs)
end
...
Same schema, different system
There are often times when different systems need access to the same table in the database. In Flatfoot, the Archer
, Spade
, and SpadeInspector
systems all require access to data from the backends
table. In earlier version of Phoenix, everyone would just refer to the same Archer.Backends
model, thereby adding yet another coupling between different systems. In Phoenix 1.3, however, we simply create customized schemas for each system.
Figure 5. Each system needing access to the
backends
table has their own schema.
While each of the three Backend
modules in Figure 5 are schemas that represent data from the same table, they only pull fields and associations according to the needs of the system (see Figure 6 below). The SpadeInspector
system only needs the module name for each backend in order to build the right configuration, while the Spade
system only needs a few fields available in case users need that data to select a backend for a particular account. We are able to deliberately encapsulate our schemas and associations within each system and minimize coupling.
Figure 6. Backends for each system.
Special considerations - associations in disparate systems
For my app, I had one situation where I needed to associate across boundaries in order to properly handle deleting a User
. Similar to the situation above, both the Clients
system and Spade
system make reference to the users
table and they both specify different has_many
associations for their unique User
schemas:
Figure 7. Same table, different associations
This all works perfectly well until someone needs to delete their account. The Clients
context needs to offer account deletion functionality, so we create a simple function within that context:
def delete_user(%User{} = user) do
Repo.delete(user)
end
When we run our tests, however, we get an error:
1) test delete_user/1 deletes associated wards from Archer (Flatfoot.ClientsTest)
test/clients/clients_test.exs:95
** (Ecto.ConstraintError) constraint error when attempting to delete struct:
* foreign_key: wards_user_id_fkey
"If you would like to convert this constraint into an error, please
call foreign_key_constraint/3 in your changeset and define the proper
constraint name. The changeset has not defined any constraint."
Our User
is still be referenced by elements of another table, so Ecto will throw that ConstraintError
. Our problem, though, is that our User
schema in Clients
does not declare the has_many
ratio with wards
table.
We could simply add the has_many :wards
and has_many :watchlists
with a on_delete: delete_all
option, which is what the Phoenix 1.3 guides suggest. But since those guides weren’t available in the RC version of 1.3.0, and I was rather obsessed with decoupling, I opted to create one central schema that contained all the associations.
I wouldn’t recommend doing this (follow the guides above), but I leave it here to show an alternate method: create a new shared system:
|-- shared | |-- schema | | |-- user.ex | | | |-- shared.ex
The system has one schema, user.ex
:
defmodule Flatfoot.Shared.User do
use Ecto.Schema
schema "users" do
has_many :sessions, Flatfoot.Clients.Session, on_delete: :delete_all
has_many :notification_records, Flatfoot.Clients.NotificationRecord,
on_delete: :delete_all
has_many :blackout_options, Flatfoot.Clients.BlackoutOption,
on_delete: :delete_all
has_many :wards, Flatfoot.Spade.Ward, on_delete: :delete_all
has_many :watchlists, Flatfoot.Spade.Watchlist, on_delete: :delete_all
end
end
And the context, shared.ex
, only one function:
defmodule Flatfoot.Shared do
alias Flatfoot.{Repo, Shared.User}
def delete_user(user_id) do
User |> Repo.get(user_id) |> Repo.delete
end
end
Now we go back and modify our Clients
context:
def delete_user(%Flatfoot.Clients.User{} = user) do
result = Flatfoot.Shared.delete_user(user.id)
if result |> elem(0) == :ok, do: {:ok, user}, else: result
end
Note that we return a Clients.User
structure instead of the Shared.User
that Flatfoot.Shared.delete_user(user.id)
returns. If we just returned the result directly, it would only be a user struct with associations, which isn’t that useful.
Conclusion
Phoenix 1.3 is a significant departure, not only from earlier iterations of framework, but also from the MVC architectures that have dominated web development recently. One can’t help but feel that Phoenix is coming into it’s own, escaping the dominating shadow of the popular frameworks before it, and offering something new, something more in line with evolving, modern web design. Additionally, Phoenix 1.3 feels more like a natural extension of an Elixir umbrella app or collection of microservices than it does a monolithic MVC app like Rails.
It takes some getting used to, but the future is bright.