Editing Message queue

{{Short description|Means of interprocess communication in software engineering}}
{{redirects|Mailbox (computing)|the file format|Mbox}}
{{Refimprove|date=May 2009}}
In [[computer science]], '''message queues''' and '''mailboxes''' are [[Software engineering|software-engineering]] [[software componentry|components]] typically used for [[inter-process communication]] (IPC), or for inter-[[thread (computing)|thread]] communication within the same process. They use a [[Queue (data structure)|queue]] for [[Message (computer science)|messaging]]&nbsp;– the passing of control or of content.  [[Group communication system]]s provide similar kinds of functionality.

The message queue paradigm is a sibling of the [[Publish–subscribe pattern|publisher/subscriber]] pattern, and is typically one part of a larger [[message-oriented middleware]] system. Most messaging systems support both the publisher/subscriber and message queue models in their [[application programming interface|API]], e.g. [[Java Message Service]] (JMS).

Competing Consumers pattern enables multiple concurrent consumers to process messages on the same message queue. <ref>{{Cite book |last=Gorton |first=Ian |title=Foundations of Scalable Systems |publisher=O'Reilly Media |isbn=9781098106034}}</ref>

==Remit and ownership==
Message queues implement an [[Asynchronous method invocation|asynchronous communication pattern]] <!-- NEED TO SYNCH THIS WITH THE SECTION BELOW. ON IT.  -->between two or more processes/threads whereby the sending and receiving party do not need to interact with the message queue at the same time. Messages placed onto the queue are stored until the recipient retrieves them.  Message queues have implicit or explicit limits on the size of data that may be transmitted in a single message and the number of messages that may remain outstanding on the queue.<ref>Dive Into Queue Module In Python. [http://linux.die.net/man/7/mq_overview Overview of POSIX message queues] </ref>

===Remit===
Many implementations of message queues function internally within an [[operating system]] or within an [[application software|application]]. Such queues exist for the purposes of that [[system]] only.<ref>Win32 system message queues. {{Cite web|url=http://msdn.microsoft.com/en-us/library/ms644927(VS.85).aspx|title=About Messages and Message Queues|work=Windows User Interface|publisher=Microsoft Developer Network|access-date=April 21, 2010|archive-url=https://web.archive.org/web/20120317065349/http://msdn.microsoft.com/en-us/library/ms644927(VS.85).aspx|archive-date=March 17, 2012|url-status=dead}}</ref><ref>Linux and POSIX message queues. [http://linux.die.net/man/7/mq_overview Overview of POSIX message queues] {{Webarchive|url=https://web.archive.org/web/20120504002336/http://linux.die.net/man/7/mq_overview |date=2012-05-04 }} at linux.die.net</ref><ref>Using Linux Message Queues. [http://www.civilized.com/files/msgfuncs.txt Linux message queue functions] {{Webarchive|url=https://web.archive.org/web/20120408091327/http://www.civilized.com/files/msgfuncs.txt |date=2012-04-08 }} at www.civilized.com</ref>

Other implementations allow the passing of messages between different computer systems, potentially connecting multiple applications and multiple operating systems.<ref>For example, the MSMQ product. {{Cite web|url=http://msdn.microsoft.com/en-us/library/ms711472.aspx|title=Message Queuing (MSMQ)|work=Network Communication|publisher=Microsoft Developer Network|access-date=May 9, 2009}}</ref>  These message queuing systems typically provide [[Resilience (network)|resilience]] functionality to ensure that messages do not get "lost" in the event of a system failure.  Examples of commercial implementations of this kind of message queuing [[software]] (also known as [[message-oriented middleware]]) include [[IBM MQ]] (formerly MQ Series) and Oracle Advanced Queuing (AQ).  There is a [[Java (programming language)|Java]] standard called [[Java Message Service]], which has several [[Proprietary software|proprietary]] and [[free software]] implementations.

[[Real-time operating system]]s (RTOSes) such as [[VxWorks]] and [[QNX]] encourage the use of message queuing as the primary inter-process or inter-thread communication mechanism.  This can result in integration between message passing and CPU scheduling.  Early examples of commercial RTOSes that encouraged a message-queue basis to inter-thread communication also include [[Versatile Real-Time Executive|VRTX]] and [[pSOS (real-time operating system)|pSOS]]+, both of which date to the early 1980s. The [[Erlang (programming language)|Erlang programming language]] uses ''processes'' to provide concurrency; these processes communicate asynchronously using message queuing.

===Ownership===
The message queue software can be either proprietary, open source or a mix of both. It is then run either on premise in private servers or on external cloud servers ([[message queuing service]]).

* Proprietary options have the longest history, and include products from the inception of message queuing, such as [[IBM MQ]], and those tied to specific operating systems, such as [[Microsoft Message Queuing|Microsoft Message Queuing (MSMQ)]]. Cloud service providers also provide their proprietary solutions such as [[Amazon Simple Queue Service]] (SQS), StormMQ, [[Solace Corporation|Solace]], and [[IBM MQ]].
* Open source choices of messaging [[middleware]] systems includes [[Apache ActiveMQ]], [[Apache Kafka]], [[Apache Qpid]], [[Apache RocketMQ]], [[Enduro/X]], [[JBoss Messaging]], [[JORAM]], [[RabbitMQ]], [[Open Message Queue|Sun Open Message Queue]], and [[Tarantool]].

Examples on hardware-based [[message oriented middleware|messaging middleware]] vendors are [[Solace Corporation|Solace]], [[Apigee]], and [[IBM MQ]].

==Usage==
In a typical message-queueing implementation, a [[system administrator]] installs and configures message-queueing software (a [[queue manager]] or broker), and defines a named message queue. Or they register with a [[message queuing service]].

An application then registers a software routine that "listens" for messages placed onto the queue.

Second and subsequent applications may connect to the queue and transfer a message onto it.

The queue-manager software stores the messages until a receiving application connects and then calls the registered software routine.  The receiving application then processes the message in an appropriate manner.

There are often numerous options as to the exact semantics of message passing, including:

* Durability – messages may be kept in memory, written to disk, or even committed to a [[database|DBMS]] if the need for reliability indicates a more resource-intensive solution.
* Security policies – which applications should have access to these messages?
* Message purging policies – queues or messages may have a "[[time to live]]".
* Message filtering – some systems support filtering data so that a subscriber may only see messages matching some pre-specified criteria of interest.
* Delivery policies – do we need to guarantee that a message is delivered at least once, or no more than once?
* Routing policies – in a system with many queue servers, what servers should receive a message or a queue's messages?
* Batching policies – should messages be delivered immediately?   Or should the system wait a bit and try to deliver many messages at once?
* Queuing criteria – when should a message be considered "enqueued"?  When one queue has it?  Or when it has been forwarded to at least one remote queue?  Or to all queues?
* Receipt notification – A publisher may need to know when some or all subscribers have received a message.

These are all considerations that can have substantial effects on transaction semantics, system reliability, and system efficiency.

==Standards and protocols==
Historically, message queuing has used proprietary, closed protocols, restricting the ability for different operating systems or programming languages to interact in a heterogeneous set of environments.

An early attempt to make message queuing more ubiquitous was [[Sun Microsystems]]' [[Java Message Service|JMS]] specification, which provided a [[Java (software platform)|Java]]-only abstraction of a client [[Application programming interface|API]]. This allowed Java developers to switch between providers of message queuing in a fashion similar to that of developers using [[SQL]] databases. In practice, given the diversity of message queuing techniques and scenarios, this wasn't always as practical as it could be.

Three standards have emerged which are used in open source message queue implementations:
#[[Advanced Message Queuing Protocol]] (AMQP) – feature-rich message queue protocol, approved as ISO/IEC 19464 since April 2014
#[[Streaming Text Oriented Messaging Protocol]] (STOMP) – simple, text-oriented message protocol
#[[MQTT]] (formerly MQ Telemetry Transport) – lightweight message queue protocol especially for embedded devices

These protocols are at different stages of standardization and adoption. The first two operate at the same level as [[HTTP]], MQTT at the level of [[TCP/IP]].

Some proprietary implementations also use HTTP to provide message queuing by some implementations, such as [[Amazon.com|Amazon]]'s [[Amazon Simple Queue Service|SQS]]. This is because it is always possible to layer asynchronous behaviour (which is what is required for message queuing) over a synchronous protocol using request-response semantics. However, such implementations are constrained by the underlying protocol in this case and may not be able to offer the full fidelity or set of options required in message passing above.

==Synchronous vs. asynchronous==
<!-- WEIRD SECTION, SHOULD THIS NOT BE PUT INTO THE ASYNCHRONOUS ARTICLE

SECTION SHOULD BE DELETED. DOES NOT ADD FACTS  -->Many of the more widely known [[communications protocol]]s in use operate [[Synchronization (computer science)|synchronously]].  The HTTP protocol&nbsp;– used in the [[World Wide Web]] and in [[web service]]s&nbsp;– offers an obvious example where a user sends a request for a web page and then waits for a reply.

However, scenarios exist in which synchronous behaviour is not appropriate.  For example, [[AJAX]] ([[wikt:asynchronous|Asynchronous]] [[JavaScript]] and [[XML]]) can be used to asynchronously send text, JSON or XML messages to update part of a web page with more relevant information.  [[Google]] uses this approach for their Google Suggest, a search feature which sends the user's partially typed queries to Google's servers and returns a list of possible full queries the user might be interested in the process of typing. This list is asynchronously updated as the user types.

Other asynchronous examples exist in event notification systems and [[Publish–subscribe pattern|publish/subscribe]] systems.
* An application may need to notify another that an event has occurred, but does not need to wait for a response.
* In publish/subscribe systems, an application "publishes" information for any number of clients to read.
In both of the above examples it would not make sense for the sender of the information to have to wait if, for example, one of the recipients had crashed.

Applications need not be exclusively synchronous or asynchronous. An interactive application may need to respond to certain parts of a request immediately (such as telling a customer that a sales request has been accepted, and handling the promise to draw on inventory), but may queue other parts (such as completing calculation of billing, forwarding data to the central accounting system, and calling on all sorts of other services) to be done some time later.

In all these sorts of situations, having a subsystem which performs message-queuing (or alternatively, a broadcast messaging system) can help improve the behavior of the overall system.

==Implementation in UNIX==
There are two common message queue implementations in [[Unix|UNIX]]. One is part of the SYS V API, the other one is part of [[POSIX]].

===SYS V===
UNIX SYS V implements message passing by keeping an array of linked lists as message queues. Each message queue is identified by its index in the array, and has a unique descriptor. A given index can have multiple possible descriptors. UNIX gives standard functions to access the message passing feature.<ref name="The Design of the UNIX Operating System">
{{cite book | title=The Design of the UNIX Operating System | author=Bach, M.J. | year=1986 | publisher=Prentice-Hall | isbn=9780132017992 | url-access=registration | url=https://archive.org/details/designofunixoper00bach }}
</ref>

;<code>msgget()</code>: This system call takes a key as an argument and returns a descriptor of the queue with the matching key if it exists. If it does not exist, and the <code>IPC_CREAT</code> flag is set, it makes a new message queue with the given key and returns its descriptor.
;<code>msgrcv()</code>: Used to receive a message from a given queue descriptor. The caller process must have read permissions for the queue. It is of two types.<ref name="Operating Systems Concepts">
{{cite book | title=Operating Systems Concepts |author=Abraham Silberschatz, Peter B. Galvin |year=1994 |publisher=Addison-Wesley |  isbn=9780201504804 }}
</ref>
:*Blocking receive puts the child to sleep if it cannot find a requested message type. It sleeps until another message is posted in the queue, and then wakes up to check again.
:*Non-blocking receive returns immediately to the caller, mentioning that it failed.
;<code>msgctl()</code>: Used to change message queue parameters like the owner. Most importantly, it is used to delete the message queue by passing the <code>IPC_RMID</code> flag. A message queue can be deleted only by its creator, owner, or the superuser.

===POSIX===
The POSIX.1-2001 message queue API is the later of the two UNIX message queue APIs. It is distinct from the SYS V API, but provides similar function. The unix man page <code>mq_overview(7)</code> provides an overview of POSIX message queues.

==Graphical user interfaces==
[[Graphical user interface]]s (GUIs) employ a message queue, also called an ''event queue'' or ''input queue'', to pass [[event (computing)|graphical input actions]], such as [[mouse click]]s, keyboard events, or other user inputs, to the [[application program]].<ref>{{cite web |last1=Cartwright |first1=Corky |title=GUI Programming |url=https://www.cs.rice.edu/~cork/newBook/node89.html |website=Rice University:Robert (Corky) Cartwright |access-date=June 27, 2020}}</ref> The windowing system places messages indicating user or other events, such as timer ticks or messages sent by other threads, into the message queue. The GUI application removes these events one at a time by calling a routine called <code>getNextEvent()</code> or similar in an [[event loop]], and then calling the appropriate application routine to process that event.<ref>{{cite book |last1=Nystrom |first1=Robert |title=Game Programming Patterns |date=2014 |publisher=Genever Benning |isbn=978-0990582908 |url=https://gameprogrammingpatterns.com/event-queue.html |access-date=June 27, 2020}}</ref>

==See also==
{{Div col}}
* [[Advanced Message Queuing Protocol]] (AMQP)
* [[Amazon Simple Queue Service]]
* [[Apache ActiveMQ]]
* [[Apache Qpid]]
* [[Celery (software)]]
* [[Gearman]]
* [[IBM Integration Bus]]
* [[IBM MQ]]
* [[Java Message Service]]
* [[MQTT]]
* [[Message-oriented middleware]], [[:Category:Message-oriented middleware|(category)]]
* [[Microsoft Message Queuing]] (known colloquially as MSMQ)
* [[NATS Messaging|NATS]]
* [[Oracle Cloud|Oracle Messaging Cloud Service]]
* [[RabbitMQ]]
* [[Redis]]
* [[TIBCO]] Enterprise Message Service
* [[Enduro/X]] Middleware platform
* [[ZeroMQ]]
{{Div col end}}

==References==
{{Reflist}}

{{Inter-process communication}}

{{DEFAULTSORT:Message queue}}
[[Category:Inter-process communication]]
[[Category:Events (computing)]]
[[Category:Queue management]]