MemNet

MemNet was devised to overcome the I/O view of networks in which the kernel is called to initiate network traffic. Instead it provides memory extension through the connection of each node through a MemNet device to a high-speed token ring network.

MemNet device

The MemNet device takes the processing of communications protocols away from the CPU. It connects to the local bus and provides access to remote data at local bus speeds. They consist of a host bus interface, a token ring interface and local memory divided into a cache and a reserved area.
The ring topology is used because:

  • Access time is limited to a absolute upper bound
  • Token ring provides explicit broadcast
  • All interfaces see the same ordering of ring transactions
  • Ring accesses are overlapped, providing a 'pipelining' effect.

Address space

The MemNet device provides the host node with access to a paged shared address space. The pages are called chunks and are 32 bytes each. Chunks are allowed to migrate between MemNet devices as required.
The sum of the reserved areas of all MemNet devices must be sufficient to store all the shared memory chunks since the system does not support disk paging.
A chunk must be in the cache area of the device attached to a node before the node can access an address in the chunk.

Retrieving pages

  • A reference to an address in the shared memory space is sent to the local MemNet device.
  • The device determines whether the reference can be satisfied locally.
  • If it cannot, a message is transmitted on the ring.
  • The device that can satisfy the request modifies the request message to include the required chunk.

Chunk faults can always be resolved in one circuit of the ring. Since this response is predictable and very fast, the device does not differentiate between accesses to data held in local cache and those requiring message transmission.
The device simply blocks the local processor as it retrieves the required data.

Page tables

A single-writer/multiple-reader protocol ensures data coherency. Each device maintains a chunk able for every chunk in the shared memory.
The attributes (or tags) maintained for each chunk include: valid, exclusive, reserved, clean and cached location.
Operations on MemNet shared memory are: read, write and an atomic read-modify-write.

  • The valid tag must be set before a chunk can be read.
    • If valid is not set the device sends off a request for a valid copy of the chunk.
  • If the provider of a chunk has exclusive set, the tag must be unset before the chunk is sent
  • A write operation can continue on a chunk if the valid and exclusive tags are set.
    • If the valid tag is set and the exclusive tag is not set, an invalidation message must be used to obtain exclusive access.
    • If neither tag is set, a message requesting a valid and exclusive copy is sent. The first receiver of this message invalidates its copy and includes a copy of the chunk.
    • The other devices with copies invalidate them as the message passes
  • Any request arriving for a chunk involved in a RMW (read-modify-write) sequence is delayed (by transmission of a wait message) until the sequence is complete.

The consistency of tags is crucial to shared data coherency. This is achieved using the network/local tags protocol.

Network/Local tags protocol

The protocol relies on the fact that ring messages are sequential. When a chunk is purged from local cache, it is transmitted on the ring and stored in the reserved memory of the first device that has the reserved bit set for the chunk. This reserved space is guaranteed to be available for the chunk.
Chunks with the clean tag set are the first to be purged on cache flush.

Disadvantages

  • The use of the chunk table renders the scheme unscalable
  • The nature of the ring network restricts the scalability of the system
  • There is no protection over access to data and no provision for fault tolerance.

Advantages

  • Location transparency is provided by the ring-based chunk request mechanism
  • Naming transparency is provided by the unique virtual addresses used to access data
Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License