net: fib_rules: fix error code for unsupported families