REST

Benjamin Erb
2013-12-12

Representational State Transfer
Distributed Architectures with HTTP

Distributed Systems #101

Distributed Systems #101

A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable.
Leslie Lamport
A collection of independent computers that appears to its users as a single coherent system.
Andrew Tanenbaum

Distributed Systems – Why?

  • more machines → more resources
  • more CPUs → more computing power
  • more storage → higher capacity

Distributed Systems – Why?

Non-functional Features

  • fault-tolerance
  • high availability
  • reliability
  • scalablity
  • performancce
  • openness

Distributed Systems – Building Blocks (1)

A program
is the code you write.
A process
is what you get when you run a program.
A message
is used to communicate between processes.
A packet
is a fragment of a message that might travel on a wire.
A protocol
is a formal description of message formats and the rules that two processes must follow in order to exchange those messages.
Source: Introduction to Distributed System Design (Google Code University)

Distributed Systems – Building Blocks (2)

A network
is the infrastructure that connects different machines via communication links.
A component
can be a process or any piece of hardware required to run a process, support communications between processes, store data, etc.
A distributed system
is an application that executes a collection of protocols to coordinate the actions of multiple processes on a network, such that all components cooperate together to perform a single or small set of related tasks.
Source: Introduction to Distributed System Design (Google Code University)

So easy! Really?


Unfortunately, distributed systems are hard!

Distributed systems you might have used already…

Networks Internet
LAN
Client/Server Applications* World Wide Web
SMTP & IMAP/POP3
P2P Applications* Bit Torrent
Skype

(*): Applications are a set of components and protocols in this context.

Programming Distributed Applications

Local application
runs on a single machine (your computer)
Distributed application
runs on multiple machines, communicates

We need a way to incorporate communication and coordination into our code.

Why not use TCP or UDP sockets directly?

Why not use TCP or UDP sockets directly?

Because it is a very bad idea most of the time. By not using existing application-layer protocols, we lose:

  • abstraction
  • knowledge and experience
  • reusability
  • interoperability
  • etc.

Don't reinvent the wheel!

Dedicated middleware systems are available to provide these features as an abstraction layer, such as

  • message queues
  • remote procedure calls
  • remote method invocations
  • web services

Message Queues

Remote Procedure Calls & Distributed Objects

"Web Services"

The idea of message systems is really cool, but they can't traverse our corporate firewalls!
Why not use HTTP for messaging?! Port 80 is already open! I like XML.
Anonymous CTO

"Web Services"

  • protocol stack: SOAP, WSDL, (UDDI)
  • provide features similar to RPC and messaging
  • XML-serialized messages
  • code generators for various languages (serialization/deserialization of SOAP messages <=> language constructs)
  • use various application-layer protocols for transport (i.e. HTTP, SMTP,…)
SOAP. Simple Object Access Protocol. Except it's not Simple, it's Complicated. And it's not about Objects, it's about RPC. So, really: CRAP
@joewalnes

"Web Services"

  • popular industry/enterprise technologies
  • appropriate for several use cases
  • very complex, tons of standards (WS-*)
  • have almost nothing to do with the actual web!

Using the Hypertext Transfer Protocol as a low-level, interchangable transport protocol for XML-encoded messages is not the same as "web".

The Challenge

Can we build distributed applications using the web like it is supposed to, embracing its ideas and concepts?

HTTP Reprise

The World Wide Web

URI + HTTP (+ HTML)

A Little Refresher

URI

Uniform Resource Identifier

Uniform Resource Identifier: Structure

 http://thehost.tld:8080/path/to/it?name=value#foo
 \__/   \______________/\_________/ \________/ \_/
   |           |            |            |      |
scheme     authority       path        query  fragment

HTTP

Hypertext Transfer Protocol

The Hypertext Transfer Protocol (HTTP) is an application-level protocol for distributed, collaborative, hypermedia information systems. It is a generic, stateless, protocol[…].
RFC 2616

Hypertext Transfer Protocol

  • request/response
  • client/server
  • stateless
  • resource-oriented
  • on top of TCP/IP

HTTP: Clients, (Caching/Reverse) Proxies, Servers

Hypertext Transfer Protocol: Request/Response

GET /foo HTTP/1.1
Host: erb.io
 

HTTP/1.1 200 OK
Date: Wed, 04 Jan 2012 17:01:59 GMT
Server: myserver/0.0.7
Last-Modified: Wed, 25 Nov 2011 12:13:15 GMT
Etag: "3ab23c-21f-12058638"
Content-Length: 11
Connection: close
Content-Type: text/plain; charset=UTF-8

hello world

HTTP Request Methods

safe idempotent cachable
HEAD ✓ ✓ (✓)
GET ✓ ✓ (✓)
PUT ✓
DELETE ✓
POST

Data Operations

Operation in SQL in HTTP
Create INSERT POST / PUT
Read SELECT GET / HEAD
Update UPDATE PUT / POST
Delete DELETE DELETE

Please note: The comparison may be flawed on closer consideration. SQL is about manipulating data sets. REST is about transforming application state changes via hypermedia, HTTP and URIs.

HTTP Response Status Codes

(very incomplete list, only some exampless)

1xx informational 100 Continue
2xx sucess 200 Ok
3xx redirection 300 Multiple Choices
301 Moved Permanently
4xx client error 400 Bad Request
404 Not Found
409 Conflict
5xx server error 500 Internal Server Error

Some Advanced HTTP Features

  • conditional requests
  • content negotiation
  • caching
  • partial requests
  • persistent connections
  • request pipelining
  • chunked encoding

HTTP: Conditional Requests (1)

GET /foo HTTP/1.1
If-Modified-Since: Wed, 18 Nov 2011 19:21:45 GMT
Host: erb.io
 

HTTP/1.1 200 OK
Date: Wed, 04 Jan 2012 17:01:59 GMT
Server: myserver/0.0.7
Last-Modified: Wed, 25 Nov 2011 12:13:15 GMT
Etag: "3ab23c-21f-12058638"
Content-Length: 11
Connection: close
Content-Type: text/plain; charset=UTF-8

hello world

Resource has been changed (here: Last-Modified mismatch).

HTTP: Conditional Requests (2)

GET /foo HTTP/1.1
Host: erb.io
If-None-Match: "103-4585efb972780;49711a471c400"
 

HTTP/1.1 304 Not Modified
Date: Wed, 04 Jan 2012 17:06:01 GMT
Server: myserver/0.0.7
Etag: "103-4585efb972780;49711a471c400"
Connection: close

Resource has not yet changed (here: ETag match).

HTTP: Content Negotiation (1)

GET /foo HTTP/1.1
Accept: text/plain, text/html
Accept-Language: de; q=1.0, en; q=0.5
Host: erb.io
 

HTTP/1.1 200 OK
Date: Wed, 04 Jan 2012 17:01:59 GMT
Server: myserver/0.0.7
Etag: "3ab23c-21f-12058638"
Content-Length: 10
Connection: close
Content-Type: text/plain; charset=UTF-8
Content-Language: de

hallo welt

Server returns german text representation, as requested.

HTTP: Content Negotiation (2)

GET /foo HTTP/1.1
Accept: application/msexcel
Accept-Language: it
Host: erb.io
 

HTTP/1.1 406 Not Acceptable
Date: Wed, 04 Jan 2012 18:21:15 GMT
Server: myserver/0.0.7
Content-Length: 22
Connection: close
Content-Type: text/plain; charset=UTF-8
Content-Language: en

Sorry, can't do that!

Content negotiation failed, server can't fulfil the request.

Hypermedia Formats

Hypermedia Representations

  • HTML
  • XHTML
  • HTML5
  • Atom

Hypermedia: Links

<a href="/foo#bar">Over there!</a>
<link rel="stylesheet" type="text/css" href="mystyle.css" />
<a rel="next" href="http://example.org/entries/3">next item</a>
<a rel="crush" href="http://social.net/hot-person">Hot!</a>

Hypermedia: Forms

<form action="http://example.org/users" method="post">
  <input name="name" type="text" value="" />
  <input name="login" type="text" value="" />
  <input type="submit" />
</form>	
POST /users HTTP/1.1
Host: example.org
User-Agent: Mozilla/5.0 (Ubuntu; X11; Linux x86_64; rv:9.0.1) Gecko/20100101 Firefox/9.0.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
Content-Length: 28

name=Benjamin+Erb&login=b_erb
Architectural Style: Representational State Transfer

REST

Representational State Transfer

REST

Introducing the Architectural Style

Client-Server

Statelessness

(Web) Applications without any state?

Almost impossible…

Here: Stateless communication!

  • server manages resource states
  • client manages session states

Statelessness

[...]each request from client to server must contain all of the information necessary to understand the request, and cannot take advantage of any stored context on the server.
Roy T. Fielding

Statelessness: Example

What's the URI of the 2nd result page of search query?

Stateful Version

http://foo.bar/query?sid=61AUOh6W1TA&page=next

  • server creates a session for each search
  • session contains search term and current page
  • "next" URI contains session ID and action (next)
  • the state of the search is hidden in the session state

Statelessness: Example

What's the URI of the 2nd result page of search query?

Stateless Version

http://foo.bar/query?term=uni+ulm&start=10

  • URI contains all information of the search state
  • search term 'uni ulm', skip the first 10 results
  • client manages search state
  • server answers request without looking up a session

Cacheability

  • improve network efficiency & scalability
  • label cacheability of each response

Layered System

  • hierarchical layers of components
  • client, servers, intermediaries
  • reverse proxies, proxy caches, load-balancers, etc.
  • integration of legacy systems

Uniform Interface: Sub-constraints

  • identification of resources
  • manipulation via representations
  • self-descriptive messages
  • hypermedia as the engine of application state

Identification of Resources

  • resource-oriented architecture
  • resources are the nouns of the application
  • resources are fine-grained entities

Examples

  • http://example.com/user/4711
  • http://example.com/customer/23/order/1/product/0735611319
  • http://example.com/search?term=foo
  • http://example.com/blog/the-post-title/comments#latest

Manipulation via Representations

  • changes of server-side resource states
  • based on changes of (altered) representations

Manipulation via Representations

Self-descriptive Messages

  • messages contain representation of resources
  • meta-data/description of representation (header)
  • usage of (custom) MIME types
  • enough information to process without additional knowledge

Hypermedia as the Engine of Application State

  • state changes based on hypermedia
  • following hyperlinks
  • submitting forms
  • accessing resources

Hypermedia: Resource Representations

  • representations should support hypermedia
  • XHTML, Atom, etc.

XML and JSON are also popular generic formats. Note however, that they don't have a standardized way of expressing links or forms.

Hypermedia as the Engine of Application State

Hypermedia as the Engine of Application State

REST Triangle

REST in Practice

REST

in Practice

RESTful Web Services

REST + HTTP

RESTful Web Services

  • web services adhering to the REST style
  • HTTP + hypermedia formats
  • e.g. leightweight APIs for mobile/web clients
REST is so trendy, let's rebrand our old HTTP API and call it RESTful from now on!
Hipster CTO

RESTful Web Services in Practice

  • REST: over-hyped term, much confusion
  • HTTP-based API vs. RESTful API
  • different types of REST "compliance"

Common mistakes

  • URIs contain actions instead of resources
  • links are hard-coded
  • messages are not self-descriptive
  • ignoring Hypermedia as the Engine of Application State

Advanced Topics and Challenges

  • API versioning
  • representation versioning
  • authentication and security in RESTful APIs
  • design of hypermedia-enabled representations
Use Case: RESTful ToDo API

Use Case

RESTful ToDo API

Source: cc-by-nc 2.0 bryankennedy on flickr

Use Case: ToDo API

  • (very) simple demo application
  • "get things done"
  • list of ToDo items
  • priorities and deadlines for entries

Domain Resources

Resource Types?

  • "homepage" (abstract resource)
  • todo item (primary resource)
  • list of todos (collection resource)
  • external documentation resource (for humans)

Domain Resources: URIs

Base URI
http://example.com/todo/
List URI
http://example.com/todo/items
Item URI
http://example.com/todo/items/{id}

We need URIs for the server implementation. Clients do not and should not need to know URIs a priori (except for the base URI of the service).

Resource Operations

  • open "homepage"
  • open documentation
  • list all todo items
  • list todo items (filtered)
  • create a new todo item
  • get a todo item
  • delete a todo item
  • change a todo item (name, deadline, status)
  • get a distinct item (latest, most urgent)

Item Resource: Uniform Interface

http://example.com/todo/items/{id}

Method Operation Req. Entity Resp. Entity
GET Read an item n/a item representation
PUT Update an item* item representation n/a
DELETE Remove an item n/a n/a



(*) If we allow clients to choose the URI of a new item by themselves, PUT can also be used for creating new items.

Item List Resource: Uniform Interface

http://example.com/todo/items

Method Operation Req. Entity Resp. Entity
GET Get a (partial) list n/a list (page)
POST Create a new item item representation n/a

Resource Representations

Option A: Generic types

  • application/json
  • application/xml
  • text/plain
  • text/html
  • application/x-www-form-urlencoded

Resource Representations

Option B: Customized (vendor-specific) types

  • application/vnd.todo.item+json
  • application/vnd.todo.list-v1+xml



We will use various representations in the following. Remember content negotiation: Clients will indicate representations they are interested in.

Resource: Documentation

  • helps developers to program clients
  • describes link relations
  • describes form types/fields
  • describes content semantics, e.g. classes in HTML
  • describes custom content types, if any

Resource: "Homepage"

  • link to documentation
  • link to distinct items
  • link to item list
  • form for new items
  • form for filtered list view

"Homepage" (here: HTML5 Representation)

<!DOCTYPE html>  
<html lang="en">  
  <head>  
    <meta charset="utf-8" />  
    <link rel="self" href="http://example.com/todo" />
  </head>  
  <body> 
    <h1>ToDos</h1>
    <h2>Navigation</h2>
    <nav>
      <ul>
        <li><a href="http://example.com/documentation/todo/" rel="help">
        Documentation</a></li>
        <li><a href="items" rel="archives">
        List all items</a></li>
        <li><a href="items/latest" rel="http://example.com/todo/rel/latest">
        Latest item</a></li>
        <li><a href="items/urgent" rel="http://example.com/todo/rel/urgent">
        Most urgent item</a></li>
      </ul>
    </nav> 
    <h2>New Item</h2>
      <-- new item form --> 
    <h2>Filtered List</h2>
      <-- filtered list form --> 
  </body>  
</html>

"Homepage" (here: HTML5 Representation)

Resource: List

  • partial list (limited length)
  • links to listed items
  • pagination (via cursor, next/previous relations)
  • two items per page

List (here: HTML5 Representation)

<!DOCTYPE html>  
<html lang="en">  
  <head>  
    <meta charset="utf-8" />  
    <link rel="self" href="http://example.com/todo/items?cursor=4" />
    <link rel="alternate" href="http://example.com/todo/items" />
  </head>  
  <body> 
    <h1>ToDos</h1>
    <h2>Navigation</h2>
    <nav> … </nav> 
    <h2>List</h2>
    <ol start="4">
      <li class="http://example.com/documentation/todo/#item">
        <a href="http://example.com/todo/items/4" rel="self">Eat some cookies</a>
      </li>
      <li class="http://example.com/documentation/todo/#item">
        <a href="http://example.com/todo/items/5" rel="self">Drink some coffee</a>
      </li>
    </ol>
    <ul>
      <li><a href="items?cursor=6" rel="next">next page</a></li>
      <li><a href="items?cursor=2" rel="previous">previous page</a></li>
    </ul>
  </body>  
</html>

List (here: HTML5 Representation)

List (here: JSON Representation)

{
   "list":[
      {
         "item":{
            "title":"Eat some cookies"
         },
         "link":{ "rel":"self", "href":"http://example.com/todo/items/4" }
         }
      },
      {
         "item":{
            "title":"Drink some coffee"
         },
         "link":{ "rel":"self", "href":"http://example.com/todo/items/5" }
      }
   ],
   "links":[
      { "rel":"self", "href":"http://example.com/todo/items?cursor=4" },
      { "rel":"alternate", "href":"http://example.com/todo/items" },
      { "rel":"next", "href":"http://example.com/todo/items?cursor=6" },
      { "rel":"previous", "href":"http://example.com/todo/items?cursor=2" }
   ]
}

Filtered Lists

  • query parameters for filters, filtered result set
  • accessible via form (using GET)
  • c.f. WHERE of SQL

Examples

http://example.com/todo/items

  • ?done=false&due_before=2012-02-15
  • ?priority=5&cursor=6

Resource: ToDo Item

  • title
  • description
  • priority
  • date due
  • done flag

ToDo Item (here: JSON Representation)

{
   "item":{
      "link":{
         "rel":"self",
         "href":"http://example.com/todo/items/4"
      },
      "title":"Eat some cookies",
      "description":"Don't forget the bright side of the dark side",
      "priority":5,
      "due":"2012-02-10",
      "done":false
   },
   "links":[
      {
         "rel":"index",
         "href":"http://example.com/todo"
      },
      {
         "rel":"help",
         "href":"http://example.com/documentation/todo/"
      }
   ]
}

Getting a ToDo Item

GET /todo/items/4 HTTP/1.1
Accept: application/vnd.todo.item+json; q=0.8, text/html
…

HTTP/1.1 200 OK
Last-Modified: Tue, 01 Feb 2012 12:45:26 GMT
ETag: "3-737060cd8"
Cache-Control: max-age=3600
Content-Type: application/vnd.todo.item+json
…

{
   "item":{
      "link":{
         "rel":"self",
         "href":"http://example.com/todo/items/4"
      }
    }
    …
}

Creating a New ToDO Item

  • using a HTML form, submit via POST
  • or: clients generates JSON, POSTs to collection resource (/items)

Creating a New ToDO Item

 <h2>New Item</h2>
  <form name="new_item" method="post" action="items" enctype="application/x-www-form-urlencoded">
    <input type="text" name="title" />  		
    <input type="text" name="description" />  		
    <input type="range" name="priority" min="1" max="5" />  		
    <input type="date" name="due" />  		
    <input type="submit" />
  </form>  				

POST /todo/items HTTP/1.1
Host: example.org
Content-Type: application/x-www-form-urlencoded
…

title=Buy+coffee&description=Out+of+caffeine&priority=5
&due=2012-02-12
HTTP/1.1 201 Created
Location: http://example.com/todo/items/13
…

Removing a ToDo Item

  • client issues a DELETE against the URI
  • server removes resource
DELETE /todo/items/4 HTTP/1.1
…

HTTP/1.1 204 No Content
…
GET /todo/items/4 HTTP/1.1
…

HTTP/1.1 410 Gone
…

Updating a ToDo Item

Remember

  • Manipulation via Representations
  • Hypermedia as the Engine of Application State

Steps

  • client GETs a representation from the server
  • client modifies represenation locally
  • client PUTs new representation to the server
  • alternative: pre-populated form (HTML)

Updating a ToDo Item

PUT /todo/items/4 HTTP/1.1
Content-Type: application/vnd.todo.item+json
…

{
   "item":{
      "link":{
         "rel":"self",
         "href":"http://example.com/todo/items/4"
      },
      "title":"Eat some cookies",
      "description":"Don't forget the bright side of the dark side",
      "priority":5,
      "due":"2012-02-10",
      "done":true
   },
   "links":[…]
}
HTTP/1.1 202 Accpeted
…

Updating: Optimistic Concurrency Control 

GET /todo/items/4 HTTP/1.1
…

HTTP/1.1 200 OK
E-Tag: "3-737060cd8"
…
PUT /todo/items/4 HTTP/1.1
If-Match "3-737060cd8"
…

HTTP/1.1 202 Accepted
E-Tag: "4-123bcd21"
…
PUT /todo/items/4 HTTP/1.1
If-Match "3-737060cd8"
…

HTTP/1.1 412 Precondition Failed
…

Get a Distinct Item

Alias URIs

Latest item
http://example.com/todo/items/latest
Most urgent item
http://example.com/todo/items/urgent
GET /todo/items/latest HTTP/1.1
…
HTTP/1.1 303 See Other
Location: /todo/items/4711
…

So what?

Results

Using HTTP as Enabling Technology

  • already solves many problems of distributed systems
  • wide-spread
  • mature, battle-tested technology

Loose Coupling & Interoperability

  • decoupled implementations
  • pick any language/framework with a HTTP library
  • clients only need limited knowledge (i.e. relations and representations)
  • reuse of existing HTTP components
  • clients can negotiate appropriate representations
  • server can change URIs and resources

Scalability

  • add more servers
  • install a load balancer in front
  • use a reverse proxy with caching support
  • integration of external services (remember the web)

Performance and Caching

  • proxies/clients can cache items
  • reduces traffic
  • efficient implementations due to caching
Conclusion

Conclusion

Recap

  • HTTP can be used for building distributed services
  • SOAP (and others) disregard principles of the web
  • REST is an architectural style that embraces the web

Recap: REST

  • resources identified by URIs
  • (hypermedia) representations of resources
  • set of verbs as uniform interface on all resources
  • hypermedia as the engine of application state

Reading Recommendations – Books

Stefan Tilkov
REST und HTTP
2011, Dpunkt Verlag, 266 pages (german)
ISBN: 978-3898647328

Leonard Richardson, Sam Ruby
RESTful Web Services
2007, O'Reilly Media, 446 pages
ISBN: 978-0596529260

Subbu Allamaraju
RESTful Web Services Cookbook
2010, O'Reilly Media/Yahoo Press, 314 pages
ISBN: 978-0596801687

Thank you!

Questions?

Contents

Backup Slides

Distributed Systems

Middleware Systems

Fallacies of Distributed Computing

  1. The network is reliable.
  2. Latency is zero.
  3. Bandwidth is infinite.
  4. The network is secure.
  5. Topology doesn't change.
  6. There is one administrator.
  7. Transport cost is zero.
  8. The network is homogeneous.
Source: http://blogs.oracle.com/jag/resource/Fallacies.html

Message-oriented Middleware

  • message queues
  • messaging between processes
  • often asynchronous and non-blocking
  • various messaging patterns

Examples

JMS, AMQP, ØMQ, etc.

Echo Server in Java

public class EchoServer
{
  public static void main(String[] args) throws IOException
  {
    ServerSocket serverSocket = new ServerSocket(4711);
    Socket socket = null;
    while ((socket = serverSocket.accept()) != null)
    {
      int b;
      while ((b = socket.getInputStream().read()) != -1)
      {
        socket.getOutputStream().write((byte) b);
      }
    }
  }
}

Echo Client in Java

public class EchoClient
{
  public static void main(String[] args) throws IOException
  {
    Socket socket = new Socket("localhost",4711);
    socket.getOutputStream().write("Hello World".getBytes());
    InputStream is = socket.getInputStream();
    InputStreamReader i = new InputStreamReader(is);
    BufferedReader r = new BufferedReader(i);
    String l;
    while( (l = r.readLine()) != null){
      System.out.println(l); 
    }
  }
}

Message Queue Example: ØMQ + node.js

var zmq = require('zeromq')

var zmqSocket = zmq.createSocket('req')

//Callback for receiving messages
zmqSocket.on("message", function(msg) {
	console.log("Received msg:" + msg);
});

//connect to message queue
zmqSocket.connect("tcp://localhost:5555");

//Sending messages
zmqSocket.send("Hello");
More about ØMQ: zeromq.org

Remote Procedure Calls & Distributed Objects

  • make remote calls look like local calls
  • access/location transparency
  • RPC: procedure calls
  • RMI: method calls + distributable objects
  • generic interface definitions, language-specific implementations
  • often blocking and synchronous, just like local calls

Examples

Sun RPC, CORBA, Java RMI, Apache thrift, etc.

RMI Example: Remote Adder with Java RMI

public interface Adder extends Remote
{
    public int add(int a, int b) throws RemoteException;
}
public class RemoteAdderImpl extends UnicastRemoteObject implements Adder
{
    protected RemoteAdderImpl() throws RemoteException
    {
        super();
    }

    @Override
    public int add(int a, int b) throws RemoteException
    {
        return (a+b);
    }
}

RMI Example: Remote Adder with Java RMI

Create and register object (JVM A):

Naming.bind("rmi://localhost/adder", new RemoteAdderImpl() );

Access remote object (JVM B):

Adder adder = (Adder) Naming.lookup("rmi://localhost/adder");
int result = adder.add(1,1);

adder.add(1,1) looks like a regular statement thanks to RMI.

Note that both JVMs can be hosted on different machines.

Contents