django - Merge/join lists of dictionaries based on a common value in Python

Question

I have two lists of dictionaries (returned as Django querysets). Each dictionary has an ID value. I'd like to merge the two into a single list of dictionaries, based on the ID value.

For example:

list_a = [{'user__name': u'Joe', 'user__id': 1},
          {'user__name': u'Bob', 'user__id': 3}]
list_b = [{'hours_worked': 25, 'user__id': 3},
          {'hours_worked': 40, 'user__id': 1}]

and I want a function to yield:

list_c = [{'user__name': u'Joe', 'user__id': 1, 'hours_worked': 40},
          {'user__name': u'Bob', 'user__id': 3, 'hours_worked': 25}]

Additional points to note:

• The IDs in the lists may not be in the same order (as with the example above).
• The lists will probably have the same number of elements, but I want to account for the option if they're not but keeping all the values from list_a (essentially list_a OUTER JOIN list_b USING user__id).
• I've tried doing this in SQL but it's not possible since some of the values are aggregates based on some exclusions.
• It's safe to assume there will only be at most one dictionary with the same user__id in each list due to the database queries used.

Many thanks for your time.

Answer 1

I'd use itertools.groupby to group the elements:

lst = sorted(itertools.chain(list_a,list_b), key=lambda x:x['user__id'])
list_c = []
for k,v in itertools.groupby(lst, key=lambda x:x['user__id']):
    d = {}
    for dct in v:
        d.update(dct)
    list_c.append(d)
    #could also do:
    #list_c.append( dict(itertools.chain.from_iterable(dct.items() for dct in v)) )
    #although that might be a little harder to read.

If you have an aversion to lambda functions, you can always use operator.itemgetter('user__id') instead. (it's probably slightly more efficient too)

To demystify lambda/itemgetter a little bit, Note that:

def foo(x):
    return x['user__id']

is the same thing* as either of the following:

foo = operator.itemgetter('user__id')
foo = lambda x: x['user__id']

*There are a few differences, but they're not important for this problem