Welcome!

Welcome to the official BlackBerry Support Community Forums.

This is your resource to discuss support topics with your peers, and learn from each other.

inside custom component

BlackBerry® Z10

Reply
Trusted Contributor
Posts: 210
Registered: ‎03-17-2013
My Device: Z-10
My Carrier: T-Mobile

SMTP problem - now documented

It's clearly the missing CRs in lines.

 

I enabled debugging in my spam filter (which "eats" messages, then examines them before processing) and this is what I'm getting in the log:

 

Apr  4 10:50:24 NewFS spamblock-sys[22755]: WARNING: Peer [192.168.1.21] had no CRs in lines

 

That's irrefutable and a clear violation of RFC 2822.

 

Now that should not result in scrambled messages, but apparently there are some email transports out there that have a serious problem with this, and in fact do scramble messages, probably because they are looking for the "CRLF" tuple as a terminator, not finding it, and running right off the end of the buffer!

 

That's hideously bad design on the part of the SMTP server writer involved, incidentally.

 

Nonetheless the rule is "be liberal in what you accept but stringent against the RFCs in what you emit", and the Z-10 at present violates this -- and the Internet RFCs for SMTP.

 

Paging Blackberry software "engineers"........

Market Information? Come read The Market Ticker!
Contributor
Posts: 46
Registered: ‎04-19-2010
My Device: BB Playbook, BB Torch 9800, BB Z10, iPhone5
My Carrier: Vodafone DE

Re: SMTP problem - now documented

My mail hoster has found the same. CRs are missing in the body.

 

http://supportforums.blackberry.com/t5/BlackBerry-Z10/SMTP-API-faulty-on-BB-Z10/td-p/2213833

 

"The mail body starts correctly with newline 2x 0d0a and ends correctly with 0d0a 2e 0d0a.

But all other 0a (\n) in the body aren't completed with 0d (\r)."

Contributor
Posts: 34
Registered: ‎09-06-2011
My Device: Priv on AT&T
My Carrier: AT&T

Re: SMTP problem - now documented

I see on your forum that at least one other person using Web.com to host their email is having the same problem. Below is a copy of the email I sent to their support department.  They replied that they are escalating it.  I hope they just don't say that it is a Blackberry problem don't bother us.

 

Here is what I sent to Web.com

 

Opened on: 4/5/2013 7:45:14 PM

Issue History on Ticket:
4/5/2013 7:45:14 PM - Online Customer:
A now known problem exists with the new Blackberry Z10 (uses OS10 - QNX) when sending email.  This is in the RFC 2822 setup and an incompatibility between configuration of the host's server and the Blackberry programming.  Either Web.com and/or Blackberry must make a change before any owners of these new BB can send mail through your servers. 

I past here a link to the explanation on the Blackberry support form.  http://supportforums.blackberry.com/t5/BlackBerry-Z10/SMTP-problem-now-documented/m-p/2286091/highli...

Please contact the Blackberry engineers and work this out.  Let me know what actions you are taking. 

I am not the only person having this problem with your servers and those from some other hosting companies. 

I can problem images of the technical messages explaining this if you want but you have to email me to get them as I cant past here. 
jgrobertson
Priv on AT&T
Formerly, Playbook, Z10 and Passport
Contributor
Posts: 34
Registered: ‎09-06-2011
My Device: Priv on AT&T
My Carrier: AT&T

Re: SMTP problem - now documented

This is what I got back from Web.com!

 

So they just pass the buck to Blackberry and put us the dumb marks who gave our money in the middle. I will proceed with a formal complaint to the US Federal Trade Commission.  The Blackberry foremost and above all else is an email machine.  All the other kdi stuff may be fun but if it does not do email it is worthless. 

 

So the compalint is for selling prodect that fails to serve the intended purpose. This is a violation of US law.

 

From Web.com

 

Solution History On Ticket: 
4/5/2013 8:20:24 PM - JOHN J.:
Dear Valued Client,

Thank you for contacting us regarding blackberry phones not working.

Please contact Blackberry to resolve your issue.

We apologize for any inconvenience this issue may have caused you.

Regards,

John J.
Web.com Unix Systems Administrator
jgrobertson
Priv on AT&T
Formerly, Playbook, Z10 and Passport
Trusted Contributor
Posts: 210
Registered: ‎03-17-2013
My Device: Z-10
My Carrier: T-Mobile

Re: SMTP problem - now documented

[ Edited ]

OK, after a fairly exhaustive set of testing on the Z-10 against my own server in all the modes I can support with it, this is what I can tell you with good authority on the SMTP protocol problems.

 

First, the Z-10 is as I documented above not sending CRs in the message body.  It is terminating lines with LF only, which is a violation of RFC2822 among others. 

 

This is not, by the way, a particularly uncommon problem; the Z-10 is not the first or only SMTP client that I've seen do this.  It is disappointing and Blackberry urgently needs to fix it, but for a well-designed SMTP server it should not be a disabling problem.

 

Next, I went through a full battery of other options - - SMTP authentication turned on and off and STARTTLS enabled and not, so as to check both authenticated and unauthenticated transactions, along with those using encryption and not using encryption.  Note that SMTP is separate from IMAP encryption; the two are completely distinct.  For the purposes of the test I disabled the prohibition that my server normally enforces on authentication (for security reasons it will not permit an authentication command over an unencrypted channel, even supposedly-secure ones, as MD5 in particular is widely considered breakable these days. For this reason you should NEVER use authentication except where your SMTP connection is encrypted.)

 

What I found is that the Z-10 has no problems with SMTP encryption or authentication, either singly or in any of the four possible combinations.  There was no combination of settings I could find that caused it to misbehave or potentially expose a password to interception.  It also properly flagged a server that has a self-signed certificate as well as one that was using a cert from a different hostname than "as-issued"; the former is very common, the latter less so.  It also had no problem negotiating a cipher against the reference OpenSSL implementations I have access to here.  Finally, not only does it honor the EHLO list of enhancements and not attempt to authenticate against a server that doesn't claim it can do so it also follows best practices and always issues the STARTTLS command before authenticating if both a username/password tuple and STARTTLS are selected.  Some clients botch this and will try to authenticate in the clear before switching to an encrypted channel, which is very bad since it exposes the username and password to interception -- the Z-10 correctly handles this circumstance and defers authentication even if the server states it will accept authentication while in plain-text mode until after it has started TLS.  Blackberry did this right; many others get it wrong.

 

One thing that is somewhat of an issue is that the Z-10 appears to always send in HTML format; I haven't found a way to tell it to send email in plain text without any embellishment as of yet.  This is something that used to be extremely important when the world of dumb terminals was still a material part of the Internet userbase, but is less-important today. Nonetheless it is an option that is preserved in most email clients and defaults on until you do something (like specify a font, bold, italic, etc) that forces some sort of rich text output, or if you are replying to a message containing it.  Blackberry should change this, as while it is rather unlikely to run into an email client that is unable to understand HTML email these days, it is certainly possible.

 

Mime encoding works fine as well, including multiple attachments in a single message -- no problems were detected there.

 

From this I conclude:

 

1. The Z-10 has a problem with message body transmission where it does not properly terminate text lines with CRLF as required, instead using a bare LF.  This is a significant but not all-that-uncommon violation of the RFCs for SMTP transmission of emails.

 

2. The Z-10 does not have a problem with SSL or TLS-encrypted SMTP transport, nor with authentication -- both work as-intended and are compliant with the appropriate RFCs, as well as best practices (specifically, never attempting to authenticate on a clear channel unless given no encryption option.)

 

3. The Z-10 appears to have no means to force a text-only email body transmission without any HTML components.  This is something that Blackberry should fix for the occasion where it is useful -- or necessary -- and should default to plain text unless the user either specifies a formatting command of some sort (e.g. a font specification, bold, etc) or is replying to a message that was originally received in HTML format.

 

I can therefore conclude with reasonable certainly if you are getting scrambled transmissions the culprit is your SMTP server, which is having an implementation-specific problem with (1) above.  Worse, such a problem is evidence of a severe security risk within your SMTP server, as the most-likely cause of this sort of misbehavior is running off the end of an internal buffer in the code.  This is exactly how hackers can and do break into systems and software that is known susceptible to this sort of fault should never be in an Internet-facing application.  Such a server needs to be immediately removed from Internet-facing service until it is fixed as it poses a risk of penetration to the organization that is operating it.

 

If you are getting multiple (or worse, infinite) retransmission attempts due to failures this is almost-certainly due to your server botching the SMTP protocol and failing to return a hard failure code for what it finds to be a mangled transmission.  Mangling the email or worse, rejecting it with other than a 5xx code is unacceptable under these circumstances but the blame for that secondary misbehavior is not Blackberry's.  If a server wishes to garf about the missing CRs it is within its rights to do so but to be compliant with Internet standards it must reject the message with permanent (5xx) series response.

 

In the interim I have also checked K9Mail (available on BlackberryWorld) and it does not suffer from the CRLF problem.  This is a potential workaround if you have an SMTP server that cannot handle the CR problem without throwing up or worse, destroying your email, until Blackberry gets the underlying problem fixed. Note that K9Mail also will interoperate with APG (if you sideload APG) and give you PGP-encrypted email support, which I personally find very useful and use for that purpose.

 

Blackberry has an urgent need to fix the CRLF botch, but they're not responsible for the scrambled transmissions or infinite retries.  That fault lies with your server's end of the transaction.

 

(For what it's worth I'm the author of a commercial spam filter package and have written SMTP protocol servers and clients for close to 20 years; the software that I run here is my own and implements the SMTP protocol "from scratch" for this purpose.  As such I have both the source code and intimate knowledge of how it works.)

Market Information? Come read The Market Ticker!
Contributor
Posts: 34
Registered: ‎09-06-2011
My Device: Priv on AT&T
My Carrier: AT&T

Re: SMTP problem - now documented

Update: 

I installed the AT&T software update 10.0.10.116 for the Z10 and now my email works with Web.com.  On some mailboxes the automatic setup put the incoming server as mail.<domain> and I had to change those to imap.<domain>.  There must be some kind of push with the Web.com host as emails come in usually within 30 to 60 seconds of sending.

jgrobertson
Priv on AT&T
Formerly, Playbook, Z10 and Passport
Contributor
Posts: 17
Registered: ‎02-07-2013
My Device: 8310, Playbook, Z30, 9900
My Carrier: Fido

Re: SMTP problem - now documented

[ Edited ]

I found a transparent workaround that works well for me so I thought I would share it here.

 

The Yahoo SMTP server handles the lack of CR well. I have a videotron account and all my outgoing emails were garbled by their SMTP server. I simply configured my videotron account on my Z10 to use the Yahoo SMTP server and that allowed all emails sent from my Z10 to get to their recipients ungarbled.

 

Here is how to modify the SMTP settings of your selected email account on the Z10:

 

  • SMTP Username: Type your Yahoo! ID (your email address without the "@yahoo.com")
  • Update SMTP Password: Type your Yahoo! password
  • SMTP Server Address: smtp.mail.yahoo.com
  • SMTP Port: 465
  • SMTP Encryption: SSL

 

Don't have a yahoo account? You can set one up in under a minute.

 

I hope BB puts out the update sooner than later but this workaround works for me for now (thanks Yahoo).

 

Sgt Wilko.

New Contributor
Posts: 4
Registered: ‎04-24-2013
My Device: z10
My Carrier: 0

Re: SMTP problem - now documented

Hi I have been having serious problems with my email since getting the z10.  Emails to clients using my email hosted by GoDaddy are displaying to them with scrambled words and strange charecters.  For example - from an email sent yesterday:

 

=/oSmiley Tongue>

I am not certain if there are any direct c=mponents

 

I read your post and wanted to make sure this is what you mean when you said that if I am getting scrambled emails the problem is #1.  If so, I will go back to GoDaddy and hope they can fix this ASAP.  Thanks

Highlighted
Contributor
Posts: 17
Registered: ‎02-07-2013
My Device: 8310, Playbook, Z30, 9900
My Carrier: Fido

Re: SMTP problem - now documented

That certainly describes the issues I had before switching to a different Smtp server. You can easily confirm this by trying the yahoo SMTP server (keeping go daddy as your POP server). Just follow the steps in my previous post.
Sgt Wilko
New Member
Posts: 1
Registered: ‎07-22-2013
My Device: Z10
My Carrier: verizon

Re: SMTP problem - now documented

Hi: Thanks for posting this. Do you know whether BB has resolved this issue with some sort of upgrade? Mine is all messed up (godaddy online, gmail works fine). Didn't quite understand how to funnel my work email through yahoo. Will doing so have the address show up as yahoo? Thanks for your help - Scott